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ABSTRACT 


'  Office  automation  has  been  credited  as  a  means  by  which  office  and 
organizational  productivity  can  be  increased.  Incorporation  of  office 
autcxTHtion  technology  can  represent  a  large  investment  for  any 
organization.  To  guarantee  the  proper  allocation  of  resources, 
government  managers  and  executives  must  utilize  evaluation  techniques 
which  insure  that  the  benefits  of  a  program  outweigh  the  costs.  Naval 
laboratories  have  been  planning  to  purchase  office  autcration 
technology  to  increase  their  productivity.  To  insure  that  the  benefits 
of  office  automation  can  be  realized  and  are  cost  effective  an 
evaluation  method  should  be  developed  v^Aiich  quantifies  increases  in 
productivity.  The  purpose  of  this  thesis  is  to  present  an  evaluation 
method  which  provides  ostensive  and  confirmable  evidence  with  regeurd 
to  productivity  changes  and  the  cost /benefit  of  office  automation.  The 
Naval  Laboratories  Technical  Office  Automation  and  Communications 
System  (NALTOACS)  Pilot  Assessment  and  the  Automated  Technical 
Information  Processing  System  (ATIPS)  at  Naval  Weapons  Center ^INWC) 
China  Lake,  California  were  used  as  the  basis  of  analysis  for  the 

z'  ^ 

development  of  the  evaluation  method.  ' 
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I .  INTRODUCTION 


The  goal  of  this  thesis  is  to  present  a  method  for  determining 
whether  office  autcmation  (OA)  increases  productivity  in  Naval 
Laboratories  and  whether  OA  is  cost  effective  for  those  laboratories. 
To  this  end  the  Naval  Laboratories  Technical  Office  Autonation  and 
Conmunications  System  (NALTOACS)  Pilot  Assessment  and  the  office 
autonation  pilot  system  at  Naval  weapons  Center,  China  Lake, 

California  have  been  analyzed.  An  overview  of  the  major  issues  of  this 
thesis  is  contained  in  this  chapter. 

This  study  addresses  three  major  issues:  productivity,  the  costs 
and  benefits  (cost/benefit)  of  office  automation,  and  the  methodology 
used  to  determine  productivity  and  its  cost/benefit.  The  first  issue, 
productivity,  is  a  measure  of  efficiency  and  effectiveness  (Greenwood, 
1984).  Efficiency  involves  ccmparing  inputs  to  outputs:  how  much  or 
how  little  input  is  required  for  acceptable  output  or  conversely  how 
much  output  is  acceptable  given  a  fixed  level  of  inputs.  Effectiveness 
involves  attainment  of  goals:  for  example,  how  a  program  or  system 
affected  profits  if  profits  were  the  organizational  goal.  The  second 
issue,  cost/benefit,  is  addressed  by  ccmparing  the  overall  benefit 
attained  from  the  system  with  the  overall  cost.  This  is  accomplished 
after  a  measure  of  office  productivity  has  been  determined.  The  third 
issue,  the  methodology  used  to  determine  productivity  and  cost/ 
benefit,  is  addressed  through  analysis  of  the  methodology'  used  by 


ccnmuni cat ions,  oonputerized  branch  exchanges,  and  data  base 
management  systans  (Levine,  1985). 

B.  POTEOTIAL  BENEFITS  OF  OFFICE  AUTOMATION 

Computer  technology  was  initially  sought  by  industry  as  a  means  of 
cutting  secretarial  and  clerical  costs.  Office  employment  is  estimated 
to  be  30%  of  the  total  labor  force  (Panko,  1985).  Greenwood  (1984) 
estimated  that  35%  of  the  office  labor  force  is  involved  in 
secretarial  or  clerical  work.  Given  these  estimates,  secretarial/ 
clerical  work  is  estimated  to  conprise  approximately  10%  of  the  total 
labor  effort  in  the  workplace.  Office  automation  has  clearly  proven 
that  it  can  increase  secretarial  efficiency  through  the  use  of  word 
processing  equipirent.  Law  and  Pereira  (1976)  have  shewn  through 
experimentation  that  the  average  secretary  can  only  type  ten  words  per 
minute  when  errors  and  disturbances  were  factored  in.  In  their 
experiments,  word  processing  equipment  use  was  judged  responsible  for 
secretaries  increasing  their  output  to  thirty  words  per  minute:  a  300% 
increase  in  output.  Additionally,  Shiff  (1974)  proposed  that 
implementation  of  cemputerized  office  equipment  could  eventually 
eliminate  typing,  transcription,  and  filing  from  office  work:  thus 
eventually  eliminating  the  secretary,  too. 

Apart  frOTi  the  reduction  in  secretarial  help,  Gottheimer  (1979) 
suggests  that  automation  of  the  office  would  significantly  reduce  the 
need  for  middle  managers.  This  can  be  done  by  making  executives  more 
efficient.  Macfarlane  (1983)  sights  several  aspects  of  OA  systems  that 
are  important  to  management  productivity.  Information  retrieval  can 
save  executives  time  by  allowing  them  to  access  perscnal  and  corporate 


information  from  their  desk.  Analytical  tools  such  as  spreadsheets  can 
be  used  to  analyze  information  once  it  has  been  retrieved.  Special 
applications  programs  such  as  graphics  allow  presentation  of 
information  in  a  manner  that  makes  reports  more  persuasive.  Electronic 
mail  allows  information  exchange  in  a  faster  and  more  inforiml  fashion 
than  norrral  mail  v^ich  leads  to  a  greater  exchange  of  ideas  and 
information. 

In  addition  to  increasing  efficiency,  OA  can  have  other  tangible 
and  intangible  benefits.  Williams  and  Jones  (1984)  list  the  following 
as  tangible  benefits: 

-  equipment  charges  eliminated 

-  temporary  help  not  needed 

-  overtime  reducticns 

-  work  force  reductions 

They  also  list  the  follcwing  as  intangible  benefits: 

-  improved  ccmnunicaticns 

-  enhanced  docuirent  appearance 

-  less  time  to  prepare  propxDsals  and  r^orts 

Additionally,  the  Honeywell  Ttechnalysis  Office  Management  Systems 
Division  conducted  a  national  survey  in  1985  to  examine  the  attitudes 
of  office  vyNorkers  who  have  access  to  OA  equipment.  That  survey 
revealed  that  90%  of  tliose  surveyed  repxsrted  that  due  to  OA  they  had 
more  time  to  devote  to  creative  activities.  Those  surveyed  also 
reported  that  OA  played  a  major  role  in  helping  them  make  better  and 
more  informed  decisions.  About  half  of  those  surveyed  stated  that  OA 
equipment  is  responsible  for  increases  in  speed,  efficiency,  and 
productivity. 

OA  has  been  identified  as  a  means  for  organizations  to  increase 
their  productivity  while  decreasing  their  costs  (Forest,  1979; 
Ctottheimer,  1979;  Williams  and  Jones,  1984). 


C,  IS  OFFICE  AOTOMATTON  REALLY  BENEFICIAL? 


Ttiere  is  a  fear  throughout  industry  that  the  potential  benefits 
extolled  by  OA  chanpions  will  not  be  realized.  The  installation  of  new 
technology  per  se  is  not  the  way  to  nake  offices  more  efficient  and 
productive,  according  to  Charles  Callahan  (EDP,  1985).  Callahan 
contends  that  office  autanation  cannot  be  justified  by  the  vague  claim 
that  OA  will  "increase  productivity"  (p.  4);  "managanent  wants  bottom 
line  payoffs"  (p.  4).  Ralph  Kleim  (1985)  wrote  that  'using  OA  systems 
does  not  necessarily  mean  a  bona  fide  increase  in  productivity. 

Kathleen  Foley  Curley  (1984)  summerizes  the  skepticism  about 
office  autcmation. 

A  real  fear,  particularly  among  line  managers,  is  that  new  office 
automation  technologies  "may  make  us  the  best  informed,  unprofitable 
ccnpany  in  our  market."  Hew  much  information  is  too  much?  What  are 
the  benefits  to  be  gained  and  hew  can  they  be  quantified?  There  is 
also  a  question  of  whether  an  investment  in  office  autcmation 
technology  is  appropriate  in  every  business  or  in  every  industry. 

(p.  37) 

The  notion  that  OA  may  not  be  appropriate  for  every  business  or  in 
every  industry  leads  to  the  following  question:  hew  can  a  business  or 
industry  determine  whetner  OA  is  appropriate  for  their  needs? 
Specifically  for  this  thesis,  how  can  Naval  Laboratories  determine 
whether  possible  OA  benefits  will  be  realized? 

D.  OFFICE  ALTOMATION  AND  NAVAL  LABO.RATORIES 

The  Navy  currently  has  eight  laboratories  that  perform  a  wide 
'-■ariety  of  research  functions.  The  goal  of  these  laboratories  is  the 
development  of  new  or  improved  equipnnent  and/or  techniq'ues  to  support 
of  the  missions  of  the  Department  of  Defense. 

As  early  as  1979  then  Secretary  of  the  Navy  (SecNav),  W.  Grahami 
Clatter,  Jr.,  stated  in  a  memorandum  to  the  Chief  of  Naval  Operations 
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that  administrative  modernization  in  the  Department  of  the  Navy  was 
required  to  cope  with  "the  volume  of  correspondence  and  administrative 
material"  (SecNav,  July  1979).  In  1982  the  Naval  Laboratories 
Tteciinical  Office  Automation  and  Conmunications  Systan  (IJALTOACS) 
program  was  established  to  coordinate  the  introduction  of  office 
s^y’stems  technology-  into  Naval  laboratories  as  a  means  of  enhancing 
productivity  (NALTOACS,  1983).  Previous  to  the  establishment  of 
NALTOACS,  three  of  the  eight  Naval  Laboratories  had  introduced  OA 
technology.  They  are: 

-  David  W.  Taylor  Naval  Ship  Research  and  Development  Center 
(NSRDC),  Bethesda,  Maryland 

-  Naval  Surface  Weapons  Center  (NSWC) 

Dahlgren,  Virginia 

-  Naval  Weapons  Center  (NIC) 

China  La)ce,  California 

David  W.  Taylor  NSRDC  was  assigned  as  the  lead  laboratory  and  was 
given  the  responsibility  for  NALTOACS.  The  Deputy  Under  Secretary  of 
the  Navy  for  Financial  Management  reaffirmed  a  requirement  frcm  the 
Commander,  Naval  Data  Automation  Ccmmand  that  an  assessment  of  the  OA 
pilot  projects  already  in  place  should  be  performed  prior  to  the 
initiation  of  any  further  OA  implementation  in  Navy  Laboratories.  The 
pilot  assessment  done  by  David  W.  Taylor  NSRDC  was  ccmpleted  without 
gathering  any  quantitative  evidence  (i.e.,  reduction  in  personnel 
costs  or  travel  costs)  relating  to  productivity  in  an  autcmated 
offi'ce.  The  study  drew  inferences  of  productivity  changes  by  gathering 
subjective  information  (i.e.,  user  opinions  and  estimates  of  hew  much 
time  per  day  the  OA  system  saved  them)  from  laboratory  OA  users.  The 
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Naval  Laboratory  personnel  were  surveyed  and  asked  to  estimate  their 


perc^ticns  of  productivity  gains  made  possible  through  the 


installation  of  the  OA  system.  The  results  of  the  pilot  asse'sment 


done  by  David  W.  Taylor  NSRDC  were  published  in  the  "Pilot  Assessment 


Final  Report"  dated  April  29,  1983. 


OFFICE  AUTOMATION  AT  NAVAL  WEAPONS  CENTER  ( NMC ) ,  CHINA  LAKE 


NWC,  China  Lake  is  the  principle  Navy  research,  development,  test 


and  evaluation  center  for  the  air  warfare,  electronic  warfare,  missile 


systems,  defense  suppression,  and  parachute  systans.  The  need  for 


office  autonation  technology  at  NWC  China  Lake  was  documented  first  in 


Proposal  for  Establishment  of  an  Automated  Technical  Information 


Processing  System  (ATIPS)  (NWC,  July  1980).  This  proposal  sighted 


"heavily  increasing  paper  workloads  along  with  reductions  in  the 


imanpcwer  currently  handling  those  workloads"  as  a  major  problem  at 


NWC.  Additionally,  delays  in  document  production  and  prdDlems  with 


copying  and  storage  of  "phenomenal  amounts  of  paper"  contribute  to  the 


waste  of  "hundreds  of  thousands  of  dollars  each  year"  (NWC,  July 


1980). 


To  handle  this  problem  two  courses  of  action  were  recormended : 


"(1)  make  each  person  at  NWC  capable  of  doing  more  work  with  less 


effort  in  shorter  periods  of  time  and  (2)  reduce  dependence  upon  paper 


as  a  medium  of  communication"  (NWC,  July  1980,  p.  2). 


The  ATIPS  Concept  Development  Paper  (NWC,  March  1982)  delineated 


the  f’onctions  necessary  of  the  ATIPS  system  to  resolve  the  problems 


brought  forth  in  the  Proposal  for  Establishment  paper  (NWC,  July 


1980).  To  be  effective  ATIPS  would  need  to  provide  the  following 


1.  Text  Processing:  word  processing,  text  entry,  document 
editing  and  revision,  special  forms/reports  production. 

2.  Records  Ntenagement:  filing,  indexing,  and  retrieval  of 
information  on  request. 

3.  Information  Processing:  analytical  models  for  analysis/ 
cdiputations/pro jections ,  file  manipulaticn,  text/data 
merging,  business  graphics. 

4.  Management  Information:  allcw  access  to  internal  and 
external  data  sources. 

5.  Ccnmunications :  provide  inter-office  connection  for 
information  exchange. 

6.  Personal  Services:  office  calendar,  daily  schedule, 
office  directory,  personal  records  handling  capability. 

Given  equipment  that  could  fulfill  the  functional  requirements 
listed  above  the  ATIPS  system  vnbs  expected  to  provide  the  follcwing 
oenefits: 

1 .  Reduce  administrative  workload  for  technical  and  administrative 
personnel . 

2.  Provide  ready  access  to  large  quantities  of  infomation. 

3.  Reduce  the  space  required  for  storage  of  informaticn. 

4.  Reduce  administrative  costs. 

5.  Reduce  the  number  and  size  of  meetings. 

6.  Reduce  the  overall  labor  cost  for  each  productive  unit  of  work. 

7.  Make  NWS  a  more  cost  effective  Research,  Development,  Test  and 
Evaluation  (RDT&E)  center. 

The  foundation  of  the  ATIPS  system  is  the  Xerox  8000  STAR  system. 
This  syston  vvas  in  its  developmental  stage  when  it  was  delivered  to 
^WC.  The  software  installation  was  provided  with  the  Xerox  systan.  The 
telecomunications  equipment  was  provided  by  Xerox,  too;  the 
telecommunications  equipment  was  also  still  in  the  developmental 
stage.  Thus,  NWC  acted  as  a  test  site  for  the  Xerox  Ethernet 
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(NALTOACS,  April  1983).  The  ATIfS  system  was  initially  operational  in 
February  1982.  The  systan  was  scheduled  to  grow  to  approximately  1500 
terminals  to  be  used  by  2500  people;  the  total  cost  of  the  system  was 
budgeted  at  $11,875  million  (NWC,  August  1981).  However,  ATIPS  funding 
was  not  continued  beyond  the  installation  of  the  pilot  system. 


F.  SUMMARY 

Evidence  indicates  that  there  are  potential  benefits  that  can  be 
realized  by  organizations  through  the  inpl anentation  of  office 
automation  systems.  The  pritrary  benefit  claimed  by  OA  is  that 
productivity  will  increase  while  cost,  in  the  long  term,  will  be 
reduced.  However,  a  survey  of  current  literature  indicates  that  there 
is  skepticism  over  the  actual  realization  of  OA  loenefits.  The 
Department  of  the  Navy  has  established  an  organization  (NALTOACS)  to 
supervise  the  implementation  of  OA  systems  in  Navy  Laboratories. 


However,  prior  to  the  iiiplementation  of  OA  systems  in  Naval 
Laboratories  evidence  should  be  presented  that  specifically 


supports 


the  claims  that  OA  increases  productivity  in  the  Naval  laboratory.  The 
Pilot  Assessment  done  by  NALTOACS  included  information  on  the 
productivity  changes  experienced  by  the  three  pilot  programs  surveyed. 
However,  tnis  inforrretion  was  gathered  in  a  qualitative  nanner.  Naval 
laboratory  personnel  were  asked  to  estimate  their  perceptions  of 
productivity  gains  made  possible  through  the  implementation  of  OA 
s'v’s  terns . 


II.  NALTOACS  PILOT  ASSESSMENT  REVIEW 

Chapter  I  gives  an  overview  of  office  automation,  its  potential 
benefits,  the  current  status  of  OA  in  Naval  Laboratories,  and  the 
status  of  the  office  autanation  syston  at  NWC  China  Lake.  A  brief 
overview  of  the  Pilot  Assessment  of  the  three  Naval  Laboratory  OA 
pilot  projects,  done  by  NALTOACS,  is  also  presented. 

This  chapter  analyzes  the  NALTOACS  Pilot  Assessment  in  detail. 

Thj-s  analysis  is  not  concerned  with  the  results  of  Pilot  Assessment 
but  rather  with  the  method  by  which  NALTOACS  performed  their 
assessment.  To  perform  an  analysis  of  the  NALTOACS  evaluation  method 
it  is  necessary  to  begin  by  addressing  evaluations  in  a  very  general 
sense:  what  they  are,  vAiat  their  purpose  is,  and  why  they  are  useful. 
Next,  the  tw  main  types  of  evaluations,  quantitative  and  qualitative, 
are  explained  including  their  strengths  and  weaknesses.  The  issue  of 
evaluation  objectivity  vs.  subjectivity  is  also  discussed. 
Additionally,  this  chapter  e3q)lains  why  the  NALTOACS  Pilot  Assessment 
is  an  inadequate  evaluation  of  the  impact  of  office  autaiBtion 
technology  in  Naval  Laboratories. 

A.  EVALUATION  DEFINITION 

What  is  an  evaluation?  Deming  defines  evaluation  as  a 
pronouncement  concerning  the  effectiveness  of  seme  treatment  or  plan 
that  has  been  put  into  effect  (Guttentag  and  Struening,  1975).  It  is  a 
"process  used  to  analyze  the  relationship  between  actual  and  desired 
effects"  (Euske,  1984,  p.  53).  Evaluation  is  the  measurement  of 
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performance  and  the  making  of  ccmparisons  based  on  those  measurements 
(Wholey,  1979).  "Evaluation  is  the  measurement  of  desirable  and 
undesirable  consequences  of  an  action  intended  to  forward  sane  goal 
that  the  actor  values"  (Riecken,  1953,  p.  1).  All  of  these  definitions 
have  measurenent  and  comparison  as  canmon  ground.  An  evaluation  my  be 
searching  for  the  effector  of  performance  or  an  evaluation  may  be 
searching  for  the  effect  that  an  action  has  on  performance;  in  both 
situations  there  is  conmonality  in  necessity  for  measurement  and 
comparison.  For  the  purpose  of  this  thesis  Riecken 's  definition  of 
evaluation  will  be  used. 

Riecken 's  definition  of  evaluation  is  very  broad  and  general.  This 
thesis  is  concerned  with  "action"  defined  as  the  installation  of  an 
office  autcmation  system.  In  the  following  sections  of  this  chapter 
this  definition  is  further  refined. 


B.  PURPOSE  OF  EVALUATION 

Anderson  and  Ball  (1978)  list  six  major  uses  and  needs  for 
evaluatioi: 

1.  To  contribute  to  decisions  about  program  installation. 

2.  To  contribute  to  decisicns  about  program  continuation, 
expansion,  or  certification, 

3.  To  contribute  to  decisions  about  program  modification. 

4.  To  obtain  evidence  to  rally  support  for  a  program. 

5.  To  obtain  evidence  to  rally  opposition  to  a  program. 

6.  To  contribute  to  the  understanding  of  basic  psychological, 
social,  and  other  processes,  (fp.  3-4) 

This  thesis  is  most  concerned  with  purposes  1,  3,  and  4.  These  three 

purposes  are  all  basically  the  same  in  substance  but  with  shades  of 


mm 


difference  in  timing  and  user/evaluator  orientation  or  bias.  The  issue 
of  timing  involves  whether  a  program  is  in  place  or  not;  purpose  one 
is  oriented  toward  a  new  program  that  is  not  yet  in  plaoe  v^tiile 
purpose  three  and  four  are  oriented  toward  a  program  that  already 
in  place.  The  issue  of  user/evaluator  orientaticn  or  bias  involves  the 
objectivity  of  the  user/evaluator;  purpose  one  is  concerned  with 
gathering  evidence  in  an  objective  manner  while  purpose  three  and  four 
are  looking  for  evidence  in  a  positive  or  negative  sense.  Funda¬ 
mentally,  all  three  of  these  purposes  can  be  condensed  into  a  single 
concept:  the  purpose  of  an  evaluation  is  to  obtain  evidence  about  the 
effectiveness  of  a  program. 

The  purpose  of  evaluation  that  I  propose  above  is  very  broad  and 
general.  Fran  this  point  forward  this  thesis  will  narrow  the  term 
"program"  used  in  the  explanation  of  evaluation  purpose  to  "the 
installation  of  office  autonation  technology."  Therefore,  for  this 
thesis  the  purpose  of  an  evaluation  is  to  obtain  evidence  about  the 
effectiveness  of  the  installation  of  office  automation  technology. 

C.  USEFULNESS  OF  EVALUATIONS 

In  general,  evaluations  are  decision  tools;  they  are  an  aid  in 
making  choices  (House,  1980).  With  regard  to  the  issue  of 
incorporating  office  automation  technology  at  Naval  Laboratories,  a 
government  manager,  executive,  or  official  could  use  an  evaluation  as 
a  aid  in  rtaking  choices  about  providing  office  autonation  equipment  to 
Naval  Laboratories.  However,  evaluations  have  a  special  usefulness  to 
government  employees  as  written  by  Franklin  and  Thrasher  (1976): 

In  an  era  of  heightened  public  concern  about  the  distribution  and 

management  of  public  funds,  at  a  time  of  dinunished  buying  power. 


when  ideology  of  a  "good  return  on  an  investment"  competes  with  more 
charitable  and  affluent  ideologies  for  ascendancy  in  the  public 
ethos,  evaluation  becomes  a  prominent  and  visible  concern  for  the 
managers  of  public  programs,  (p.  2) 

In  a  democracy  ultimate  accountability  is  to  the  public;  the  public, 
by  direct  expression  and  through  its  elected  representatives,  is 
increasingly  daranding  a  rendering  of  that  account,  (p.  3) 

According  to  Joseph  tiJl-ioley  (1979),  those  in  charge  of  government 
programs  must  make  a  countless  number  of  decisions  under  tremendous 
tiiTie  constraints,  often  with  little  reliable  information  on  program 
performance.  He  writes  that  the  key  to  better  decisions  and  better 
government  programs  is  the  establishment  of  realistic  and  measurable 
objectives  and  measures  of  program  performance  and  the  use  of  program 
performance  information  to  bring  about  changes  in  program  activities 
that  will  ultimately  enhance  program  performance.  In  short,  Wholey 
advocates  using  program  evaluation  as  the  major  decision  aid  for 
government  managers. 

In  summary,  an  evaluation  of  office  automation  systems  at  Naval 
laboratories  is  the  measurement  of  desirable  and  undesirable 
consequences  due  to  OA  installation.  The  purpose  of  an  evaluation  of 
Naval  Laboratory  OA  systems  is  to  obtain  evidence  about  their 
effectiveness.  This  evidence  is  useful  to  government  managers  and 
executives  who  are  responsible  for  Naval  Laboratory  Office  Autcmation 
Programs  because  evaluations  enhance  program  performance  in  an  era 
when  maximum  efficiency  and  effectiveness  is  demanded  by  the  public. 

D.  TYPES  OF  EVALUATION 

Michael  Quinn  Patton  (1978)  writes  that  there  are  two  main  types 
of  eiBhuation:  q’uantitative  and  qualitative.  He  explains  that  these 


two  types  of  evaluation  are  due  to  the  two  basic  paradigns  that 
dominate  evaluative  techniques.  Ttie  first  paradigm,  which  he  says  is 
the  doninant  paradigm  in  evaluation,  espouses  the  hypothetico- 
deductive  methodology.  This  is  more  ocrmonly  known  as  the  scientific 
method;  it  assumes  that  quantitative  measuransnt,  experimental  design, 
and  statistical  analysis  is  the  epitcme  of  good  science  (Patton, 

1978).  The  secorvd  paradigm,  which  Patton  calls  the  alternative 
paradigm,  espouses  the  holistic- inductive  methodology.  This  paradigm 
relies  on  qualitative  data,  holistic  analysis,  and  detailed 
description  derived  from  close  contact  with  the  targets  of  study 
(Patton,  1979). 

In  what  situation  is  it  appropriate  to  use  each  of  these  methods? 
Patton  (1979)  writes  that  the  hypxDthetico-deductive  (quantitative) 
approach  aims  at  performance  and  prediction  while  the  holistic- 
inductive  (qualitative)  approach  aims  at  understanding.  Kuhn  wrote: 
"the  most  deeply  held  values  concern  predictions:  they  should  be 
accurate;  quantitative  predictions  are  preferable  to  qualitative  ones" 
(Patton,  1978,  p.  184).  Patton  (1978)  contends  that  for  predictions: 
"qualitative  analyses  in  general  have  little  legitimacy  beyond  certain 
limited  exploratory  situations"  (p.  211).  However,  Patton  (1978)  does 
cx)ncede  that  there  are  situaticns  where  quantitative  measurement  is 
impossible  (as  in  cases  where  an  attempt  to  measure  "goodness"  is 
made)  and  where  the  qualitative  method  is  the  only  practical  method. 

He  also  states  that  to  attain  the  goals  set  for  some  evaluations  it 
rray  oe  necessary  to  'use  the  qualitative  method. 

Analogous  to  the  two  tvpes  of  evaluation,  quantitative  and 
qualitative,  is  ei'aluation  ob'jectivity  and  sub]ectivity .  EXaluation 


objectivity  is  the  ability  to  evaluate  without  the  influence  of 
personal  feelings  (House,  1980).  Evaluation  subjectivity  is  evaluation 
by  personal  feelings.  Objective  measures  are  distinguished  fran 
subjective  measures  by  the  presence  or  absence  of  ostensive  indicators 
(Guttentag  and  Struening,  1972).  Ostensive  indicators  are  hard  data: 
data  that  is  external  to  an  evaluator/observer,  data  that  can  be 
specified  and  measured  using  some  agreed  upon  measurement  tools. 

Patton  (1978)  writes: 

Qualitative  methodology  and  a  responsive,  illuminative  approach  to 
evaluation  most  frequently  stimulate  charges  of  subjectivity  -  a 
label  regarded  as  the  very  antithesis  of  scientific  inquiry. 
Objectivity  is  considered  the  sine  qua  non  of  the  scientific  method. 
To  be  subjective  means  to  be  biased,  unreliable,  and  ncnrational. 
Subjective  data  implies  opinicn  rather  than  fact,  intuiticn  rather 
than  logic,  impression  rather  than  confirmation.  Scientists  are  to 
eschew  subjectivity  and  ma]<e  their  wor)<  "objective  and  value  free", 
(p.  216) 

This  quotation  from  Patten  should  not  be  construed  that  opinion, 
intuition,  and  impressioi  are  mutually  exclusive  cf  fact  and  logic. 

The  key  point  is  that  opinion,  intuition,  and  impression  are  not 
confirmable.  Evaluations  should  seek  evidence  that  is  ostensive  and 
confirmable. 

Which  of  these  two  methodologies  should  be  chosen  to  evaluate  a 
program?  Patton  (1978)  wrote  that  different  )d.nds  of  problems  require 
different  t>pes  of  research  methodology.  NALTOACS  "was  established  to 
coordinate  the  introduction  of  office  systems  technology  into  the  Naval 
Laooratory  as  a  means  to  enhance  productivity."  (NALTOACS,  1983,  p.  1) 
Ihe  objectives  for  the  NALTOPCS  Pilot  Assessment,  were  to  gather 
information  relating  to  OA  system  planning,  implementation,  and 
benefits  analysis  (NALTOACS,  1983).  The  first  tw  objectives, 


gathering  planning  and  inpl orientation  inforration,  are  qrjestions  of 
understanding;  these  objectives  can  best  be  attained  tiirough  use  of 
the  qualitative  methodology  (Patton,  1979).  The  last  objective,  the 
benefits  analysis,  is  a  question  of  predicting  performance.  This  is 
the  most  important  objective  of  the  Pilot  Assessment  because  it 
determines  whether  the  project  is  worthwhile:  no  matter  hew  vv«ell 
planned  or  implemented  a  systoti  is  the  bottom  line  is  cost/benefit;  if 
the  benefits  do  not  exceed  the  costs  then  the  syston  is  not  worth 
planning  or  implementing.  To  perform  a  cost/benefit  analysis  the 
quantitative  methodology  should  be  anploysd;  it  is  the  method  that 
results  in  a  true  botton  line  of  dollars  and  cents  and  it  is  the 
method  that  provides  confirmable  results. 

The  Pilot  Assessment  of  the  Naval  Laboratories  that  had  pilot 
office  automation  systems  in  place  was  an  evaluation  using  the 
qualitative  trethodology.  Appendix  A  contains  a  sample  of  the 
questionnaires  and  interview  sheets  used  by  NALTOACS  to  perform  their 
evaluation.  These  questionnaires  clearly  sought  the  opinions  and  the 
perceptions  of  the  system's  users  and  managers.  This  thesis  does  not 
take  issue  with  the  selection  of  this  methodology  to  gather 
information  about  OA  system  planning  and  implementation.  However,  this 
methodology  is  inadequate  to  perform  a  cost/benefit  analysis  because 
it  does  not  return  ostensive  data.  Also,  there  are  biases  inherent  in 
the  tools  of  the  qualitative  methodology,  questionnaires  and 
interviews,  that  cause  the  results  evaluations  using  this  nethodology 
to  be  suspect.  These  biases  are  discussed  in  the  next  secticn. 


E.  QUALITATIVE  METHODOLOGY  SOURCES  OF  INVALIDITY  IN  THE  NALTOACS 

PILOT  ASSESSMENT 

To  this  point  information  has  been  presented  about  the  identity, 
purpose,  and  usefulness  of  evaluations  in  general.  Also,  the  two  basic 
types  of  evaluations  have  been  presented  with  their  perspective 
strengths  and  weaknesses  with  an  argunent  why  the  quantitative 
methodology  is  preferable  to  the  qualitative  methodology  for  the 
performance  of  a  benefits  analysis.  It  has  been  established  that  the 
Pilot  Assessment  conducted  by  NALTOACS  used  the  qualitative 
methodology:  the  questionnaires  and  personal  interviews  are  subjective 
in  nature  (see  Appendix  A);  this  method  invited  opinion,  intuition, 
and  impression  which  is  difficult  if  not  impossible  to  confirm  as 
fact.  This  section  addresses  the  biases  and  sources  of  invalidity  that 
result  fron  surveys  and  questionnaires  themselves.  The  NALTOACS  Pilot 
Assessment  is  sited  to  exemplify  the  biases  that  explained  in  this 
section. 

Webb  et  al.  (1966)  list  three  broad  sources  of  bias  and 
invalidity  when  the  qualitative  methodology  is  used: 

1.  errors  that  rray  be  traced  to  those  being  studied, 

2.  errors  that  cane  fron  the  investigator, 

3.  errors  associated  with  sampling  imperfections. 

It  is  beyond  the  scope  of  this  thesis  to  address  the  latter  two 

sources  of  evaluation  invalidity.  However,  errors  that  may  be  traced 

to  those  being  studied,  called  the  reactive  measurement  effect  by  Webo 

et  ai.  (1966),  are  discussed  in  detail. 

The  most  understated  risk  to  valid  inter ^^^retat ion  is  the  error  pro¬ 
duced  by  the  respondent .  Ev'en  when  he  is  well  intentioned  and  coop- 
•:riti'.'e,  ‘:he  research  subject's  knowledge  tnat  he  is  participating 
in  a  sonolarly  search  may  confound  the  investigator's  data.  Four 


classes  of  this  type  of  error  are:  awareness  of  being  tested,  role 
selection,  measurement  as  a  change  agent,  and  response  sets.  (p.  13) 

Of  the  four  of  reactive  measurement  effect  errors  listed  above,  only 

awareness  of  being  tested,  role  selection,  and  response  sets  are 

applicable  to  this  thesis;  they  are  discussed  below. 

1 .  The  Awareness  of  Being  Tested 

The  awareness  of  being  tested  has  been  called  the  "reactive 

effect  of  measuronent"  bias  by  Campbell  (1957).  Selltiz  et  al.  (1959) 

wrote: 

If  people  feel  that  they  are  "guinea  pigs"  being  experimented  with, 
or  if  they  are  being  "tested"  and  mast  make  a  good  impression,  or  if 
the  method  of  data  collection  suggests  responses  or  stimulates  an 
interest  the  subject  did  not  previously  feel,  the  measuring  process 
may  distort  the  experimental  results,  (p.  97) 

This  does  not  mean  that  all  subjects  that  are  aware  that  they  are 

part  of  a  test  will  give  inaccurate  responses.  Hcwever,  the 

probability  of  bias  is  high  in  any  study  in  which  a  respondent  is 

aware  that  he  is  the  subject  of  a  test  (Vtebb  et  al . ,  1976). 

The  subjects  in  the  NALTOACS  Pilot  Assessment  were  definitely 

aware  that  they  v^ere  part  of  an  experiment.  The  "NALTOACS  PILOT 

ASSESSMENT  USER  INTERVIEW  CHBCKLIST  PACKAGE"  (see  Appendix  A)  contains 

the  following  instructions: 

The  data  being  collected  by  this  survey  is 
required  to  conduct  the  "Pilot  Assessment" 
directed  by  the  Deputy  Under  Secretary  of  the 
Navy  (Financial  Management)  and  the  Ccmmander, 

Naval  Data  Autonation  Canmand. 

Please  take  this  opportunity  to  candidly  express 
your  opinions  and  needs  so  that  the  overall 
planning  for  office  autonation  (OA)  in  the  Naval 
Laboratories  can  directly  benefit  from  them.  Also, 
please  feel  free  to  add  any  inputs  which  you  may 
have  and  which  are  not  covered  by  the  checklist. 
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These  instructions  could  influence  the  subject  to  give  responses  so 
that  they  make  a  "good  Impression"  as  described  above  by  Selltiz. 

There  are  at  least  two  reasons  for  this.  First,  the  survey  has  been 
directed  by  the  Deputy  Under  Secretary  of  the  Nai^  who  is  in  autiiority 
far  above  the  respondents  in  the  chain  of  ccmrmnd.  This,  coupled  witn 
the  cuestionnaire  requirement  for  respondents  to  sign  the  survey  may 
nave  resulted  in  an  unwillingness  by  the  respondents  to  give  res:x)nses 
that  would  "create  waves".  The  second  reason  is  related  to  the  first 
in  that  the  collector  of  tiie  surv'ey  is  from  Washington  and  is,  in  some 
sense,  in  authority  over  the  installation  where  the  respcndents  work 
and  tne  OA  eqaiprrent  is  installed.  There  might  be  a  tendency  to  give 
positive  responses  to  avoid  "telling  on  oneself"  as  an  organization 
a.nd  incurring  scrutiny  from  above. 

2 .  .Role  Selection 

Role  selection  is  another  way  that  a  respondents  awareness  of 
research  may  produce  bias.  This  bias  may  not  be  a  defensive  reaction 
or  dishonesty  but  rather  a  specialized  selection  of  a  "proper 
beriavtLor"  by  the  respondent  (Webb  et  al.,  1976).  The  contention  oy 
^^feob  et  al .  here  is  that  a  person  who  is  singled  out  by  an 
experimenter  forces  the  subject  into  a  role  defining  decision,  a 
decision  as  to  what  t^^pe  of  person  should  the  respondent  be  to  answer 
the  questions. 

The  "NALTOACS  PILOT  ASSESSMEOT  USER  CHECKLIST  PACKAGE"  (see 
Apiendix  A)  gives  the  respondent  an  opportunity  to  select  a  role  even 
;>ufore-  tne  respondent  takes  the  survey.  After  the  instructions  a  "u-ser 
profile"  section  asxs  the  respondent  for  the  following  inf onration : 


USER  PROFILE 


Date: 


Name: _  Code: _ 

Telephone: _  Bldg: _ 

How  would  you  classify  your  job  responsibilities: 


CoTunand  and  Support 

_  Executive 

_  Manager 

_  Supervisor 

_  Professional 

_  Clerical 

Other 


Technical 

_  Manager 

_  Scientist 

_  Systems  Engineer 

_  PrograiTiTEr/Analyst 

Other 


The  implication  of  this  is  that  the  respondents  could  "play  a  role", 
as  suggested  by  Webb  et  al . ,  and  answer  questions  not  as  they 
personally  feel  but  as  they  feel  an  executive,  a  manager,  or  a 
professional  should  answer  the  questions. 

3 .  Response  Sets 

Response  sets  deal  with  the  way  that  questions  are  asked.  For 
example,  Sletto 's  experirrentation  led  to  his  assertion  that 
respondents  will  endorse  a  statement  more  frequently  than  they  will 
disagree  with  its  opposite  (1937).  Campbell  wrote  that  the  direction 
of  wording  can  definitely  influence  the  respondents  in  q^iestionnaires 
and  inter/iews  (1965).  Webb  et  al.  demonstrated  that  respondents 
generally  have  a  preference  for  strong  statanents  rather  than  moderate 
or  indecisive  statements  (1966).  They  also  danonstrated  that  sequences 
of  questions  asked  in  a  very  similar  format  produced  stereotyped 
answers,  such  as  a  tendency  to  select  a  rightham  or  lefthand 
response.  Further,  they  proposed  that  decreasing  attention  produces 
biases  from  the  order  of  presentation. 

The  NALTOACS  PILOT  ASSESSMENT  USER  INTERVIEW  CHECKLIST  PACKAGE 


Lsee  Appe.ndix  A)  has  questions  that  could  cause  response  set  errors. 


There  are  questions  that  ask  the  respondent  to  choose  an  answer  from 
amcng  a  list  of  applicable  answers.  However,  74%  of  the  first  answers 
listed  were  responses  that  were  positive  for  office  autonation  in 
general  (see  Appendix  A,  p.  68).  For  exaiTiple: 

10.  Please  check  the  appropriate  responses: 

CALENDAR 

_  Essential  _  Available  _  Use  frequently 

_  Nice  to  have  _  Not  available  _  Use  occationally 

_  Do  not  need  _  Do  not  know  _  Do  not  use 

if  available 

I  have  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 

_  Adequate  to  my  needs 

_  Must  be  improved, 

hew  so 


"Essential",  "Available",  "Use  frequently",  "Adequate  to  my  needs", 
and  "Easy  to  use"  all  possess  a  positive  connotation  while  "Do  not 
need",  "Do  not  know  if  available",  "Do  not  use",  "Must  be  inproved" , 
and  "Difficult  to  use"  all  possess  a  negative  connotation.  There  are 
five  pages  of  questions  similar  to  the  one  above  covering  the  various 
capabilities  of  the  pilot  office  automation  systans. 

11.  How  do  you  rate  the  overall  responsiveness  of  the 
system  in  operation? 

_  Fast  (under  2  sec)  _  Slow  (5-10  sec) 

_  Adequate  (2-4  sec)  _  Too  slow  (10  sec  +) 


17.  In  what  way  has  tlie  system  assisted  you  in  the 
preparation  of  products? 

___  Get  the  job  done  with  less  effort,  how  much  so?  _ % 

_  Better  quality  of  end  products,  hew  so? 

_  Fewer  typos 

_  Better  appearance 

_  More  timely 

_  Other  (specify)  _ 
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Easy  to  use 
Difficult  to  use, 
how  so 


Seventy  percent  of  the  questionnaire  consisted  of  questions 
that  vs/ere  also  asked  in  a  similar  format.  The  questionnaire  vas  twenty 
pages  long;  fourteen  of  these  pages  (see  Appendix  A,  questions  10  and 
18)  were  similar  in  format.  IXie  to  the  length  and  repetition  of  the 
question  format  it  is  possible  that  stereotyped  answers  were  given  as 
Webb  et  al.  proposed  above.  The  sheer  length  of  this  questionnaire 
brings  into  play  the  Webb  et  al .  assertion  that  decreasing  attention 
can  produce  biases. 

F.  SUMMARY 

This  chapter  has  examined  the  NALTOACS  Pilot  Assessment 
qualitative  evaluation  methodology.  This  methodology  was  determined  to 
be  appropriate,  in  principal,  to  fulfill  the  first  two  objectives 
(gathering  information  about  QA  system  planning  and  inplanentatioi )  of 
the  NALTOACS  Pilot  Assessment.  Hcwever,  this  methodology  was 
considered  inadequate  to  fulfill  the  most  important  objective,  a 
costA^enef its  analysis.  Ihere  are  two  reasons  why  the  qualitative 
methodology  is  inadequate  for  a  oost/benefits  analysis.  First,  results 
are  based  on  opinions  \4iich  are  not  ostensive  and  confirmable  in 
nature.  For  example,  the  NALTOACS  Pilot  Assessment  made  quantitative 
estimates  of  productivity  gains  due  to  OA  incorporation  without 
performing  any  quantitative  analysis;  inferences  were  drawn  fron 
questionnaires  and  interviews  -  inferences  of  fact  based  on 
respcndents '  opinions.  Specifically,  the  Pilot  Assessment  (NALTOACS 
1983'  estimates  a  productivity  gain  of  8  to  12%  and  a  savings  per  year 
of  1,681  to  2,415  man-years  across  the  seven  Naval  Laboratories  (p. 
49'.  This  thesis  does  not  contend  the  figures  that  NALTOACS  presents 


as  inaccurate  per  se.  However,  there  is  no  ostensive  data  here  nor  a 
nnethod  of  conf irmaticn.  Thus,  the  conclusions  drawn  are  suspect. 


The  seccnd  reason  that  the  qualitative  methodology  can  result  in 
erroneous  evaluation  conclusions  is  that  the  tools  of  tliis  methodology 
(questionnaires  and  interviews)  can,  in  and  of  themselves,  bias  the 
respondents.  Thus,  even  if  the  methodological  approach  was  appropriate 
to  perform  a  benefits  analysis  the  results  would  still  be  suspect  due 
to  these  errors  and  biases. 
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III.  PROPOSED  OA  BENEFITS  ANALYSIS  METHOD 


Chapter  II  critically  analyzes  the  NALTOACS  Pilot  Assessment 
method.  It  begins  with  a  discussion  of  evaluations  in  general, 
explains  the  strengths  and  weaknesses  of  the  two  main  types  of 
evaluation  methodologies,  and  discusses  evaluation  objectivity  and 
subjectivity.  Then  the  NP^TOACS  Pilot  Assessment  methodology  is 
examined;  its  inadequacy  to  attain  its  most  important  objective  (an  OA 
benefits  analysis)  is  specifically  addressed. 

This  chapter  proposes  a  detailed  method  through  which  an  OA 
benefits  analysis  can  be  performed.  This  is  done  by  studying  the  ATIPS 
program  located  at  Naval  Weapons  Center  China  Lake,  California. 

As  previously  stated  in  Chapter  II,  the  corner  stone  of  the 
quantitative  methodology  is  the  scientific  method.  The  scientific 
method  dictates  that  quantitative  measurement  and  catparative  analysis 
is  the  essence  of  reliable  and  valid  evaluation  (Patton,  1978). 
Measurement  is  the  key,  without  it  there  can  be  no  comparative 
analysis.  The  first  task  in  tailoring  the  quantitative  evaluation 
method  into  a  specific  application  must  be  to  select  a  measurement 
method  that  is  consistent  with  the  objectives  of  the  evaluation.  The 
evaluation  objective  that  is  the  focus  of  this  chapter  is  an  OA 
cost/benefit  analysis.  Therefore,  a  method  to  measure  the  costs  and 
benefits  of  an  OA  system  must  be  devised. 

Prior  to  the  installation  of  any  program  or  syston  there  should  be 
perceived  need  for  that  program/system.  This  will  be  the  starting 
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point  for  the  OA  system  performance  msasurenent  criteria.  Wholey  wrote 

that  an  evaluation  of  any  program  or  system  must  docunent  the  extent 

to  which  management  has  defined  the  measures  of  resource  inputs, 

program  activities,  outcanes,  and  impacts  (1979). 

Tnere  are  two  parts  to  the  question  of  measurability:  (1)  the 
indicator  of  achievement  and  (2)  the  means  of  verification.  It  is  a 
standard  part  of  program  evaluation  methodology  tnat  both  indicators 
and  means  of  verification  be  developed  as  part  of  any  evaluaticn. 
(SchiTU-dt  et  al.,  1977) 

The  implication  of  these  statements  is  that  performance 
meas'urement  can  and  snould  oe  done  using  management  criteria  as  a 
oasis  for  evaluation.  This  type  of  approach  to  program  evaluation  is 
)<nown  as  "goal  achievement"  (Franklin  and  Thrasher,  1976).  The 
advantages  to  the  goal  achievement  approach  is  that  it  allows  the 
implementor  to  be  the  control  for  the  study  and  it  insures  that  the 
eval'uation  is  measurable  and  relevant  to  the  implementor  ’s  original 
problems  (Franklin  and  Thrasher,  1976).  This  approach  is  particularly 
useful  in  the  evaluation  of  innovative  programs  (Rossi  and  Freerran, 
1982).  Office  automation  systems  in  Naval  Laboratories,  specifically 
the  ATIPS  system,  are  innovative.  Therefore,  this  thesis  uses  the  goal 
acfaevonent  approach  in  its  measurement  of  the  performance  of  the 
ATIPS  program  at  NWC  China  Lake. 

Rossi  and  Freeman  (1982)  write  that  the  evaluation  of  an 
innovative  program  should  include: 

1.  Identifying  and  describing  the  problem  of  concern, 

2.  Identifying  the  program  and  delineating  the  objectives  developed 
to  solve  the  problan, 

3.  Designing  an  impact  model, 

4.  Designing  an  evidence  gathering  technique. 


This  chapter  follows  this  basic  outline.  The  outline  contains  terms 
and  concepts  that  may  not  be  intuitively  obvious;  these  terms  and  the 
concepts  in  each  step  are  explained  as  they  are  addressed. 

A.  PROBLEM  IDEOTIFICATION  AND  DESCRIPTION 

The  following  paragrapn  is  taken  fron  the  ATIPS  Concept 

Development  Paper  (NWC,  Nbrch  1982): 

The  Naval  Wteapons  Center  (NWC),  China  Lake,  faces  heavily  increasing 
administrative  workloads  along  with  reductions  in  tne  manpower 
currently  handling  those  v/orkloads.  NWC  personnel  work  in  an  almost 
exclusively  manual  administrative  environment.  What  is  needed  is  a 
more  efficient  mediodology  for  handling  the  necessary  administrative 
tasks  of  Center  personnel,  (p.  1) 

An  Inspector  General's  inspection  of  NWC  in  Nferch  1980  reported 
that  manpower  ceilings  had  resulted  in  an  increase  in  personnel 
workload  and  a  decrease  in  the  quality  of  work  (NWC,  July  1980). 
Through  the  1970s  the  workforce  at  NWC  had  been  reduced  twenty  percent 
due  to  billet  restrictions  while  the  total  budget  and  workload  had 
increased  twenty  percent  (NWC,  July  1980). 

The  problem  that  faced  management  at  NWC  China  Lake  was  how  to  do 
more  work  with  less  people  without  negatively  effecting  the  quality  of 
the  end  product. 

B.  PROGRAM  AND  OBJECTIVES 

In  response  to  the  problems  listed  above,  four  alternative  courses 
of  action  were  investigated;  the  alternative  that  prarused  the  hignest 
return  on  investment  in  the  shortest  period  of  time  was  the 
coordinated  develofxnent  of  an  Automated  Technical  Information 
Processing  System  (ATIPS)  (NWC,  Aug'ust  1981).  As  discussed  previously, 
ATIPS  is  an  office  autorration  system  that  uses  a  central  computer  and 
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remote  access  termirals  to  provide  a  variety  of  functions  to  tne 

users.  Among  these  functions  are  (NALTOACS,  1983): 

word  processing  electronic  mail 

document  processing  electronic  calendar 

business  graphics  form  processing 

Tne  objectives  of  ATIPS  were  delineated  in  tne  Concept  Dev^elopmeii 
Paper  Auc'ust  1981).  These  oojectives  are  listed  below: 

1.  Reduce  admunistrative  workload  for  technical  and  adininistrativ'e 
personnel . 

2.  Reduce  tne  amount  of  space  requirea  to  store  information. 

3.  Reduce  tne  coses  for  taping,  copying,  transmitting,  filing,  and 
retrieving  information. 

4.  Reduce  the  number  and  size  of  meetings. 

5.  Reduce  tne  labor  cost  for  each  productive  unit  of  work. 

6.  Make  NWC  a  more  cost-effective  Research,  Development,  Test,  and 
Evaluation  Center. 

7.  Provide  access  to  other  NWC  computers  through  a  single  terminal. 

8.  Provide  a  direct  carmunication  route  aiixjng  all  personnel  wno 
require  it. 

9.  Proi’ide  ready  access  to  large  quantities  of  textual  information. 
Another  oojective  of  the  system  (Objective  10)  that  was  not 
cammunicated  in  the  Concept  Development  Paper  but  was  ccmmunicated  to 
t.ne  author  oy  the  f'unctional  manager  i.nvolved  tne  quality  of  the 
deliverables  that  the  Center  produced;  ATIPS  was  to  increase  the 
quality  of  the  reports,  manuals,  briefings,  and  documents  that  the 
Center  produced. 

Gr^ienwood 's  (1984)  contention  that  productivity  is  a  measure  of 
•  f  f  Lci’  nuy  and  ef  fecti'ieness  can  be  ap^jiied  to  the  stated  ATIPS 
'Ui I’.-f'-s .  Efficiency  is  the  car,{nrison  of  inputs  to  outputs;  tno 
:•  w  r  innuts  :>?r  unit  of  outiot  (or  the  ur-nt-er  (r^tput  :er  ..nit  of 


input)  Identifies  an  increase  in  efficiency.  The  first  seven 
objectives  listed  for  ATIPS  are  q-jestions  of  efficiency;  each  of  these 
objectives  seeks  to  maximize  outputs  wnile  mani-Tazing  inputs.  The  last 
three  objectives  are  not  concerned  directly  with  efficiency  (although 


tnat  rr'a\'  oe  a  side  ►^ff 


rather  tnese  objectives  are  q^uestions  of 


effectiveness,  of  hew  well  eacn  '^arxer  and  t.he  Center  can  achieve  its 
overall  goal. 

The  basic  oojtcctive  of  ATIPS  can  oe  reduced  to;  increasing 
jjersonal  and  Center  productivity  througn  increasing  both  efficiency 
ana  effectiveness. 


C.  IMPACT  .VDDEX 

An  irrpact  model,  wnich  is  similar  to  a  logical  model  (Wholey, 
1979),  IS  an  attempt  to  translate  notions  regarding  tne 
:nodifications  and  controls  of  a  condition  into  a  hypotheses  on 
which  action  can  be  based  (Rossi  and  Frearan,  1982).  Tnere  are  two 
nypotheses  that  should  be  addressed: 

1.  t.ie  causal  hypothesis, 

1.  the  inter/ention  hypot.hesis  (Rossi  and  Freeman,  1982). 

The  causal  hypothesis  is  a  hypothesis  arout  the  influence  of  one  or 
more  processes  on  the  condition  that  tne  program  seexs  to  modify.  It 
is  tne  relationship  between  the  program  or  systan  inputs,  activities, 
and  outcares  (Wholey,  1979).  For  NWC  China  Lake  the  causal  nypotnesis 
is:  The  availability  of  only  manual  administrative  tools  counhai  w;tn 
in  increasing  workload  and  a  decreasing  workforce  is  r-'s: onsi.:i e  Mr 
vwir  .roductivity  at  NWC  China  Lake. 

Isin.g  tnis  causal  hypothesis  as  a  oase,  tne  r'nnaindnr  nu  m'V' 
upa  .-i  :'odel  can  oe  developed  through  t;ie  i nt' ‘Pn-;r.t  mh  nytctiiosM 


(Rossi  and  Freeman,  1982).  The  intervention  hypothesis  is  a  statement 

that  specifies  the  relationship  between  a  program,  what  is  going  to  be 
done,  and  the  causal  hypothesis.  In  the  case  of  NWC  China  Lake  there 

are  three  variables  (manual  administrative  tools,  increasing  workload, 

and  decreasing  work  force)  in  the  causal  hypothesis  that  influence  the 

outcar.e  (productivity).  The  only  means  by  which  to  change  the  outcome 

IS  by  changing  one  of  the  three  variables.  However,  it  was  not 

possible  for  the  Center  to  hire  more  workers  or  to  refuse  projects  and 

lignten  its  workload.  Therefore,  the  first  two  variables  are 

constants.  The  only  variable  in  which  change  was  feasible  was  the  use 

of  manual  administrative  tools.  ATIPS  was  purchased  to  replace  manual 

administrative  tools  with  automated  tools.  The  goal  was  to  increase 

worker  productivity.  Therefore,  the  intervention  h^’pothesis  is:  Office 

automation  technology'  increases  overall  productivity  at  NWC  China 

Lake. 

The  iT.pact  or  logical  model  provides  a  clear  view  of  the 
relationships  oetween  the  initial  conditions  and  problems,  the 
inter.’ening  program,  and  the  outcomes. 

LV I  PENCE  GATHER  ING/MEASuRE^.T  .MODEL 

T'nj  e'.’idence  gathering  technique  that  is  used  in  an  evaluation  is 
,enced  cy  the  basic  evahuation  methodology  and  the  evaluative 
i:  pnoach.  The  oasic  evaluation  metnodology  that  tnis  proposed  OA 
must  Ijfinefits  analysis  metnoG  uses  is  the  q'uantitative  metnod 
descnoed  in  cnapter  two.  Tne  e'/al -.ative  approacn  tnat  this  profxisfid 
OA  cost  oer.u'fits  analysis  metnod  uses  is  the  goal  acmeMaaent  a.jproacn 
descriDec  earlier  in  tnis  cnapter.  The  irrtOiication  of  usini-  tne 


ciJ^ntitative  methodology  and  the  goal  achievanent  approach  is  that  the 
evidence  gathering  technique  should  yield  ostensive  data  for 
comparative  analysis  by  utilizing  the  program's  objectives  and  tiie 
program  nanagement 's  projected  performance  measurement  criteria. 
Tnereiore,  the  project  ATIPS  performance  criteria  and  tne  program's 
objectives  are  examined  prior  to  the  selection  of  an  evidence 
gathering  teclmique.  Onoe  the  evidence  gathering  tecnnique  is 
determined  the  measures  of  program  performance  are  refined  fran  tne 
projected  program  performance  criteria  and  the  program's  objectives. 
Following  that  refinement  evidence  gathering  technique  can  be 
Lmpiemented . 

1 .  Projected  ATIPS  Performance 

Nld  contracted  The  Mattox  Group,  a  consulting  firm  in 
Pasadena,  California,  to  study  a  portion  of  the  warkers  at  the  Center 
for  the  purpose  of  identifying  potential  opportunities  to  increase  the 
efficiencn^-  and  effectiveness  of  the  Center's  managerial,  technical, 
and  support  personnel  (Mattox,  February  1981).  The  study  consisted  of 
three  parts.  First,  a  "shadowing"  was  conducted  of  each  member  of  a 
pilot  group  for  8  to  16  nours.  Second,  250  representatives  frcxr.  all 
derjartments  were  interviewed.  Third,  interview  data  was  correlated 
w'ltr.  tne  "shadow"  data  (see  Appendix  3  for  Mattox  Group  evaluation 
data).  Tne  Mattox  Group  study  concluded  tnat  an  office  automation 
H/st-arr,  at  t.he  Center  could  nave  a  tremenaous  impact  and  potentially 


sa'.'e  .variagers,  technical  and  support  staff  20*.  of  tneir  time. 


Appendix  B  for  the  figures  and  calculations ) .  The  analysis  indicated 
that  ATIPS  would  result  in  a  $5-7  M  personnel  cost  avoidance  in  its 
first  year  of  operation.  Tnis  figure  would  increase  as  ATIPS  grew  to 


Its  full  size  and  tne  life  cycle  cost  avoidance  for  personnel  coula 
reacn  S151  (MC,  Aug^ust  1981). 

The  second  area  of  quantitative  measureirent  criteria  involves 
tra'uei  .  The  >attox  Group  estimated  t.oat  a  15%  reduction  in  travel 
requir-enients  -would  be  possible  tnrough  the  use  of  ATIPS  cannunication 
capaoilities  (Mattox,  Feoruary  1981).  NVC  personnel  were  averaging 
■'000  off  Center  trips  per  year  at  an  average  cost  of  $1,200  per  trip 
(MIC,  Aug-jst  1981).  This  meant  tnat  ATIPS  could  save  $420  K  in  its 
first  year  of  operation  and  $8.82  .M  over  its  life-cycle  (see  Appendix 
B).  In  addition  to  the  reduction  in  the  cost  of  travel  the  personnel 
cost  avoidarice  that  resulted  from  the  decreased  travel  time  v^as 
calculated.  This  would  result  in  a  $70  K  cost  avoidance  in  the  first 
■year  of  ATIPS  operation  and  a  $1.4  M  cost  avoidance  over  the  life- 
cycle  (NVC,  August  1981)  (see  Appendix  B). 

The  project  group  also  estimated  that  50%  of  the  Center's 
costs  for  .raintenanoe  of  office  equipment  would  be  avoided  due  to  less 
•wear  and  decreased  usage  of  copiers,  ty'pewriters ,  and  otner  office 
equi’rrent.  Tnis  would  result  in  a  $200  K  savings  the  first  year  of 
ATIPS  operation  and  a  $4.2  M  savings  over  the  life-cycle  of  the 
system.  (NWC,  Aug'ust  1981)  (see  Appendix  B). 

2 .  ATIPS  Prog'ra.m  Objectives 

The  goal  achievement  method  of  performance  measurem.ent 
requires  tnat  tne  evidence  gathering  tecnniq-ue  for  a  proc;rarr,  .do  basea 
^yon  tne  management 's  meas'uranent  criteria  and  the  program,  's 


objectives  and  goals  (Franklin  and  Thrasher,  1976).  As  presented 
previously,  the  ATIPS  program  objectives  are: 


1.  Reduce  administrative  workload  for  technical  and  administrative 
personnel . 

2.  Reduce  the  costs  for  typing,  copying,  transmitting,  filing,  and 
retrieving  information. 

3.  Reduce  the  labor  cost  for  each  productive  unit  of  work. 

4.  Reduce  the  number  and  size  of  meetings. 

5.  Reduce  the  amount  of  space  required  to  store  information. 

6.  Provide  access  to  other  NIC  ccrputers  through  a  single  terminal. 

7.  Provide  a  direct  cotrounication  route  among  all  personnel  who 
require  it. 

8.  Provide  ready  access  to  large  quantities  of  textual  inforration. 

9.  Increase  the  quality  of  the  deliverables  proauced  by  the  Center. 

10.  Make  NWC  a  more  cost-effective  Research,  Development,  Test,  and 
Evaluation  Center. 

3.  Evaluation  Technique  Selection 

The  projected  ATIPS  performance  criteria  and  the  ATIPS  program 
objectives  are  similar  in  that  they  both  suggest  that  the  introduction 
of  ATIPS  will  result  in  either  reductions  or  increases  of  costs, 
workloads,  cost-effectiveness,  and  quantities  of  textual  information, 
for  example.  This  suggests  that  measurements  made  before  the 
introduction  of  ATIPS  could  be  conpared  to  measurements  made  after  the 
introduction  of  ATIPS;  this  comparison  would  result  in  indication  of 
benefits  realized/not  realized  due  to  the  introduction  of  ATIPS.  This 
type  of  evidence  gathering  technique  is  known  as  a  cuasi-experin'ental 
time  series  design  (Franklin  and  Thrasher,  1976).  Wholey  (1979) 
describes  tne  quasi-experimental  time  series  design  of  evidence 
la  tner  iTig . 


.  .  .  Dhe  evaluator  measures  outcome  variaoles  several  times  before, 
and  several  times  after,  participants  are  exposed  to  the  program. 
Program  effectiveness  is  estimated  by  canparing  ihe  post -prog ram. 
meas ■or aments  witn  the  pre-program  ,Teasurements.  The  evaluator  plots 
the  time  series  data;  tnen  the  evaluator  and  other  knowledgeaole 
people  examine  the  data,  consider  possible  alternative  explanations 
for  the  observ’ed  results,  and  attampt  to  draw  conclusions  about  the 
effectiveness  of  the  program,  (p.  160) 

Tnis  e'.’idence  gamering  technique  fits  the  with  the  projected  ATIPS 

;;jer f ormance  criteria  and  t.he  objectives  listed  for  tne  ATIPS  program. 

Tne  quasi-experimental  time  series  design  is  the  evidence  gathering 

tecnniq^ue  rhat  this  proposed  evaluation  .Tiethod  uses. 

4 .  Ref  i.ner.'ient  of  Program  Performance  .Pleasures 


Goal  acnievement  is  the  evaluative  approach  used  for  the  OA 
oenefits  evaluation  method  proposed  oy  this  thesis.  As  stated  earlier, 
t.ne  goal  achievanent  approach  requires  thar  the  program  objectives  and 
tine  projected  progran  perfonrance  measures  oe  used  as  the  basis  for 
the  evaluation.  This  means  that  the  selection  of  specific  performance 
rreas'ures  should  involve  the  program's  objectives  and  projected 
performance  measures. 

The  ATIPS  program  oojectives  are  listed  on  page  9  of  this  chapter. 
These  10  objectives  can  oe  canbined  into  five  groups  of  objectives: 

1 .  Reduce  the  cost  of  administration  by  reducing  the  workload  and 
reducing  the  cost  per  unit  of  output. 

2.  Allow  access  through  computer  terminals  to  multiple  computers, 
large  text  libraries,  and  otner  personnel. 

3.  Reduce  tne  number  of  meetings. 

4.  Reduce  storage  spaoe  needed  for  information. 

3.  Increase  the  c-uality  of  the  worx  produced. 

The  ATIPS  program's  projected  perfcrrrance  criteria  are 
explained  on  pages  7  and  8  of  tnis  chapter.  Tnese  r>2rformance  criteria 
list  tnree  areas  of  oerf ormance  measurement: 


1.  personnel  cost  avoidance, 

2.  reduction  of  meetings  requiring  travel  away  from  NWC  China  Laxe, 

3.  reduction  in  the  cost  of  maintenanoe  on  office  equipment. 

How  do  tile  objectives  relate  to  the  pierforrrance  measurement  criteria 
estaolisned  oy  management  prior  to  the  introduction  of  ATIPS?  There 
apriears  to  be  only  one  direct  relationship,  that  is  eetween  the 
OD^ective  to  reduce  meetings  and  the  projected  performance  criteria  to 
reduce  travel  due  to  meetings.  However,  tnis  projected  performance 
measuranent  does  not  include  meetings  that  are  neld  at  NWC  C.nina  La<e. 

The  cost  avoidance  performance  meas'uremait  is  only  loosely 
related  to  the  objectives  of  cost  reduction  of  adranistration  and 
reduction  of  the  niimber  of  meetings.  Cost  avoidance  as  a  perfonrance 
measure  does  not  lend  itself  to  true  measure.ment  after  the  ATIPS 
program  is  in  place.  This  is  in  contrast  to  the  other  two  projected 
performance  criteria  whicn  do  lend  t.nemselves  to  measurenEnt. 
Therefore,  t.ne  author  views  cost  avoidance  as  more  of  a  goal  of  the 
ATIPS  system  rat.ner  tnan  a  performance  .meas'ure  for  the  program 's 
stated  goals. 

The  question  reiraining  is  wnat  t'^pes  of  measures  of 

’Erfcrra.nce  s.nould  be  used  to  provide  evidence  about  the  benefits 

cotai.ned  from  the  incorporation  of  tne  ATIPS  system?  This  thesis  can 

cr.ly  offer  suggesticns  as  to  the  appropriate  measures  of  performance 

.xrcause  to  select  these  meas^ires  requires  the  involvement  of  the 

earn  :u-.7ent  (Rossi  and  FrGe.Tan,  19H2).  Vvholey  (1979)  \Na:ote: 

I'.'.o  uey  to  useful  perfonrance  ;nonitoring  is  agreenent  between  tne 
:vul.;ator  and  tne  intended  user  on  tne  events  to  be  .ronitored,  tne 
: -eas ures  to  oe  used  in  tne  monitoring,  and  tne  intended  uses  of  the 
r-  suItiTiu  information,  (p.  137) 


This  information  leads  the  autnor  of  this  tnesis  to  sucgest 


appropriate  performance  measures  for  each  of  the  oojectives  of  the 
ATI PS  program.  Each  of  tne  onjective  groupings  on  page  9  are  addressed 
in  order. 

To  measure  the  effect  of  ATIP3  on  the  volume  of  paperwork  at 
Nl'^C  (Objective  1)  paper  itself  should  be  measured.  A  ccmparison  of 
paper  usage  should  yield  a  reduction  in  usage  after  ATI  PS  was 
introduced.  This  meas'ure  should  be  weighted  to  account  for  the 
differences  in  project  load  that  NWC  experiences  d’uring  tne 
meas’ureirient:  periods. 

To  measure  the  tne  effect  of  ATI PS  on  the  cost  of 
administrative  work  (Objectives  2  and  3)  a  ccmparison  of 
administrative  output  and  cost  should  oe  made.  Administrative  output 
is  not  synonyiTOUs  with  total  paper  usage.  Output  is  tne  sum  of  all  tne 
manuals,  reports,  briefings,  presentations,  letters,  and  the  like  that 
^'C  Cliina  Lake  produces  during  a  period  of  time.  Cost  is  the  total 
equipment  rraintenance  and  labor  costs  during  the  same  period  of  time. 
Labor  costs  are  particularly  difficult  to  measure  because  of  the 
concept  of  cost  avoidance.  It  is  unlikely  that  there  will  be  any 
savings  in  personnel  labor  costs  per  se.  However,  tne  concept  of  cost 
avoidance  implies  that  personnel  will  ccxplete  tasks  in  a  more  tbrely 
fashion.  The  implication  of  this  statement  is  that  personnel  snould  be 
aoie  to  perform  more  tasks  in  the  same  unit  of  time.  Therefore,  since 
M\"C  is  not  expected  to  reduce  its  worx  force,  to  detect  a  chancre  in 
laoor  cost  :x;r  unit  of  work  will  require  .measuring  worKload  and  staff 
size:  the  outxit  should  go  up  while  tne  lanor  force  remains  constant. 


To  measure  the  effect  that  ATI PS  has  on  the  number  and  the 


size  of  meetings  (Objective  4)  the  ninber  of  off  station  trips  made 
due  to  meetings  can  be  measured  and  compared  (as  suggested  by  the 
projected  perfomrance  meas'urarent  criteria).  Also,  historical  records 
could  be  used  to  determine  the  number  of  on  station  meetings  required 
before  ATIPS;  the  number  of  meeting  required  since  the  installation  of 
ATIPS  could  be  measured  and  a  comparison  made. 

A  measure  of  the  amount  of  space  required  to  store  informaticn 
(Objective  5)  could  involve  researching  supply  records  to  obtain 
information  concerning  the  purchase  of  storage  units  before  and  after 
the  incorporation  of  ATIPS. 

Measurement  of  the  access  to  other  computers  tnrough  ATIPS 
terminals,  measurement  of  ccnmunications  capabilities,  and  the  amount 
of  textual  information  available  through  an  ATIPS  terminal  (Ctojectives 
6-8)  can  all  be  measured  directly  without  research. 

Meas’urement  of  the  quality  of  the  deliverables  produced  by  the 
center  (Objective  9)  before  and  after  ATIPS  could  be  done  by  direct 
ccxrparison.  Copies  of  deliverables  produced  before  ATTPS  incorporation 
could  be  compared  with  deliverables  that  are  similar  in  nature  but 
produced  after  ATTPS  incorporation.  Ttie  judges  of  the  quality  of  the 
deliverables  could  be  a  composite  team  of  professionals  from  both 
outside  and  within  NWC  China  Lake.  Stufflebeam  et  al.  (1974)  wrote 
that  professional  judgenent  is  an  effective  evaluation  tool  vdien 
measurement  criteria  are  difficult  to  define  and  quantify. 

Professicnal  judgement  is  based  on  the  evaluators  experience  and 
expertise.  The  criteria  of  the  quality  measurement  should  be  agreed 
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upon  by  the  professionals  prior  to  the  actual  performanoe  of  the 
evaluaticn . 


The  final  objective  is  an  objective  that  encompasses  all  of 
the  others  and  includes  the  purpose  of  this  proposed  evaluation 
method:  making  NIC  more  cost  effective.  Measurement  of  the  other 
objectives  will  provide  evidence  for  this  objective.  This  is  where  the 
issue  of  cost/ltenef it  is  resolved.  The  measures  described  above 
produce  figures  of  relative  savings.  These  relative  savings  should  be 
totaled  and  corr.pared  to  the  cost  of  the  system.  If  the  savings  total 
is  greater  than  the  cost  of  the  system  then  the  system  is  worth 
purchasing;  it  is  worth  purchasing  for  all  the  Naval  Laboratories. 

E.  SUMMARY 

This  chapter  describes  an  method  of  evaluating  the  impact  of 
office  automation  technology  in  Naval  Laboratories.  The  Naval  Weapons 
Center  China  Lake,  California  is  the  subject  for  a  case  study;  by 
utilizing  information  from  Nli)C  an  evaluation  method  is  presented  using 
the  qrualitative  methodology  and  the  goal  achievanent  evaluative 
approach.  The  goal  of  the  evaluation  method  is  to  provide  ostensive 
data  for  the  performance  of  a  costA^enefit  analysis. 


IV.  SUMMARY  AND  RECOMMENDATIONS 


This  thesis  presents  a  method  for  determining  the  productivity 
effects  and  the  oost /benefit  of  office  automticn  in  Naval 
Laboratories.  Formulation  of  this  method  included  analysis  of  the 
NALTOACS  Pilot  Assessment  and  an  analysis  of  the  pilot  office 
automation  system  (ATIPS)  at  NWC  China  Lake,  California. 

Chapter  I  presents  a  definition  of  office  automation  and  provides 
evidence  that  organizations  can  realize  both  tangible  and  intangible 
benefits  through  the  incorporation  of  OA  technology.  The  primary 
benefit  claimed  for  OA  is  increased  productivity.  However,  a  survey  of 
current  literature  indicated  skepticism  ov^-r  the  realization  of 
benefits  due  to  OA.  The  chapter  concludes  with  an  overview  of  the 
NALTOACS  Pilot  Assessment  and  the  ATIPS  syston  at  NWC  China  Lake. 

Ghapter  II  analyzes  the  NALTOACS  Pilot  Assessment  method  in 
detail.  A  qualitative  method  was  used  to  perform  the  NALTOACS  Pilot 
Assessment.  The  chapter  provides  evidence  showing  that  this 
methodology  was  inappropriate  for  two  reasons.  First,  the  results  of 
the  qualitative  evaluation  are  based  on  opinions  which  are  not 
cstensive  and  ccnfirmable  in  nature.  Second,  the  qualitative 
methodology  can  produce  erroneous  results  due  to  the  biases  inherent 
in  this  evaluative  paradigm. 

Chapter  III  presents  a  method  of  evaluating  the  impact  of  office 
automation  in  Naval  Laboratories,  a  method  based  on  a  quantitative 
method.  This  eval'uation  method  incorporates  the  goal-achievar.ent- 
ev'aluative  approach  and  uses  the  ATIPS  system  at  NWC  China  Lake  as  a 


subject  of  a  case  study.  The  goal  of  the  evaluation  nEthcd  is  to 
provide  ostensive  data  for  the  perfonr.ance  of  a  oost/benefit  analysis. 
The  purpose  of  using  the  goal-achievement -evaluative  approach  is  to 
insure  that  the  goals,  objectives,  and  performance  criteria  developed 
during  project  initiation  are  used  in  the  project  evaluation. 

The  ATIPS  office  automation  program  was  estimated  to  cost  nearly 
$12  million.  Incorporation  of  OA  systems  at  all  eight  Naval 
laboratories  cculd  cost  more  than  $100  mil lien.  One  performance 
criterion  for  the  investment  of  public  funds  is  that  the  trustees  of 
those  funds  maximize  the  return  on  the  investment;  this  is 
responsibility  of  the  position  and  demanded  by  the  public  (Franklin 
and  Thrasher,  1976).  To  ensure  that  goal  of  maximizing  return  on 
investment  is  achieved,  the  implementation  of  evaluation  techniques  is 
required.  These  evaluation  techniques  should  produce  measurable  and 
confirmable  results. 

The  NALTOACS  Pilot  Assessirent  was  a  qualitative  evaluation  of 
office  automation  in  Naval  Laboratories.  The  evaluation  results  were  a 
compilation  of  the  opinions  of  the  sur\-eyed  respondents;  inferences 
about  benefits  and  productivity  gains  due  to  OA  were  made  based  on 
those  opinions.  The  actual  productivity  results  were  not  measured  nor 
confirmed  nor  did  they  appear  to  be  measurable  or  confirmable.  Hence, 
the  NALTOACS  Pilot  Assessment  does  not  provide  inforrration  wiiich  is 
useful  in  determining  the  program,  s  return  on  investiment . 

The  evaluation  method  proposed  uy  tnis  thesis  troauces  results 
that  are  ooth  .measurable  and  conf irTr.able.  Purt;ier,  tne  data  productxi 
bv  the  evaluation  method  insures  tnat  the  objectives  of  time  smsteir:  are 


considered  in  its  evaluation  and  that  the  program's  cost  effectiveness 
is  determined. 

NALTOACS  was  established  to  coordinate  the  introduction  of  OA 
systems  as  means  to  enhance  productivity  in  Naval  Laboratories 
(NALTOACS,  1983).  The  reconmendation  of  the  author  is  to  perform  an 
evaluation  of  an  operating  Naval  Laboratory  OA  systan  using  the  nethod 
described  in  this  thesis.  This  will  provide  ostensive  data  which  will 
indicate  whether  office  automation  does  indeed  enhance  productivity 
and  whether  that  enhancement  is  cost  effective. 
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APPENDIX  A 


NALTOACS  PILOT  ASSESSMENT  USER  INTER'/IEW  CHECKLIST 


I*.  .r,  •'< y , 
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The  data  being  collected  by  this  survey  is  required  to  conduct 
the  "Pilot  Assessment"  directed  by  the  Deputy  Under  Secretary  of 
tJie  Navy  (Financial  Management)  and  the  Commander,  Naval  Data 
Automation  Command. 

Please  take  this  opportunity  to  candidly  express  your 
opinions  and  needs  so  that  the  overall  planning  for  office 
automation  (OA)  in  the  Naval  laboratories  can  directly  benefit 
from  them.  Also,  please  feel  free  to  add  any  Inputs  which  you 
may  have  and  which  are  not  covered  by  the  checklist. 

Thank  you  for  your  cooperation  and  assistance. 


Project  Manager 


Telephone : 


Date: 

Code ; _ 

Bldg: _ 


How  would  you  classify  your  job  responsibilities: 


Technical 


_  Executive 

_  Manager 

_  Supervisor 

_  Professional 

_  Clerical 

_  Other  _ 


Manager 
Scientist 
Systems  Engineer 
Prog rammer /Analyst 
Other  _ 


Did  you  uae  other  keyboard  devices  in  your  office  before  the 
pilot  OA  was  implemented? 


No _  Yes _ 

Typewriter 
Computer  Terminal 
WP  Workstation 
Personal  Computer 


Still  use _ 

_  Typewriter  _ 

_  Computer  Terminal  _ 

_  WP  Workstation  _ 

_  Personal  Computer  _ 


2.  Do  you  have  a  personal  computer  at  home? 
No _  Yes _ 


3.  Do  you  have  your  own  pilot  OA  workstation? 

No _ ,  how  far  away  is  the  workstation  you  share?  _  ft. 

Yes _ ,  with 

_  Dedicated  Letter  Quality  Printer 

_  Shared  Letter  Quality  Printer  How  far  away? _  ft. 

_  Dedicated  Dot  Matrix  Printer 

_  Shared  Dot  Matrix  Printer  How  far  away? _  ft. 

_  Shared  Plotter  How  far  away? _  ft. 

_  Local  Storage 


4.  Do  you  need  your  own  dedicated  OA  workstation? 
No _  Yes _ 


5.  Do  you  or  would  you  use  it  as  a  terminal  for  programming? 

No _  Yes _ 

6.  How  long  have  you  been  an  active  user  of  the  pilot  OA  system? 

_  Less  than  1  month  _  Over  6  months 

_  Less  than  3  months  _  Over  12  months 

_  Less  than  6  months  _  Over  18  months 


7.  Do  you  feel  that  you  were  adequately  prepared/trained  to  use 
the  OA  system? 

No _  Yes _  Did  not  attend  training  sessions _ 


8.  Do  you  feel  the  system  provides  enough  OA  self-help  features 
to  enable  you  to  use  it  with  ease? 

No Yes  ,  for  frequently  used  capabilities 

No Yes  ,  for  infrequently  used  capabilities 


▼ 


s 


>.  Do  you  f*«l  th«  pilot  project  staff  provides  adequate  user 
assistance? 

No _  Yea _ 


LO.  The  following  ace  general  descriptions  of  the  basic  OA 
functional  capabilities  included  in  sone  pilot  OA  systems. 
After  each  is  a  series  of  questions  pertaining  to  your 
knowledgei  use  and  assessment  of  them.  (Please  check  the 
appropriate  response/cesponses. ) 

CALENDAR  -  Allows  a  user  to  maintain  an  appointment  calendar, 
both  putlic  and  private.  It  provides  the  functions  for 
adding,  changing,  deleting,  searching  and  viewing  the 
entries. 

_ Essential  _ Available  Ose  frequently 

_ Nice  to  have  _ Not  available  _ Ose  occasionally 

_ Do  not  need  _ Do  not  know  if  _ Do  not  use 

available 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 

_ Adequate  to  my  needs  _ Easy  to  use 

_ Must  be  improved,  _ Difficult  to  use, 

how  so  how  so 


TICKLER  Notifies  the  user  on  a  given  day  and  time  that  a 
certain  action  or  task,  previously  entered  by  the  user,  needs 
to  be  performed  that  day. 

_ Essential  _ Available  _ Use  frequently 

_ Nice  to  have  _ Not  available  _ use  occasionally 

_ Do  not  need  Do  not  know  if  _ Do  not  use 

available 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


_ Adequate  to  my  needs 

_ Must  be  improved, 

how  so 


_ Easy  to  use 

_ Difficult  to  use, 

how  so  _ 


TRT.RPHQNE  -  Provides  the  user  with  a  standard  telephone 
message  pad  for  taking  telephone  messages  for  other  users  on 
the  system.  The  message  will  be  sent  to  the  recipient 
electronically. 

_ Essential  _ Available  _ Use  frequently 

_ Nice  to  have  _ Not  available  _ Use  occasionally 

_ Do  not  need  _ Do  not  know  if  _ Do  not  use 

available 


dL 


S  V 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


.Adequate  to  my  needs 
.Must  be  improved, 
how  so  _ 


_ Easy  to  use 

_ Difficult  to  use, 

how  so  _ 


PIRECTPBY  “  Provides  a  convenient  way  for  looking  up  the 
telephone  numbers  of  associates  that  are  frequently  called. 


.Essential 
.Nice  to  have 
.Do  not  need 


.Available 
.Not  available 
.Do  not  know  if 
available 


_ Use  frequently 

_ Use  occasionally 

_ Do  not  use 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


.Adequate  to  my  needs 
.Must  be  improved, 
how  so  _ 


.Easy  to  use 
.Difficult  to  use, 
how  so  _ 


BULLETIN  BOARD  -  Announces  important  events  or  information  of 
general  interest  to  all  or  a  specified  group  of  users. 
Announcements  may  refer  to  a  longer  explanatory  record  that 
can  be  retrieved  by  interested  parties. 


.Essential 
.Nice  to  have 
.Do  not  need 


.Available 
.Not  available 
.Do  not  know  if 
available 


.Use  frequently 
.Use  occasionally 
.Do  not  use 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


_ Adequate  to  my  needs 

_ Must  be  improved, 

how  so  _ 


.Easy  to  use 
.Difficult  to  use, 
how  so  _ 


ELECTRONIC  MAIL  -  Replaces  the  inter-office  memoranda  and 
other  informal  correspondence.  It  allows  the  user  to  scan 
his  mail  box,  to  read  mail,  to  reply  to  the  mail,  and  to  file 
and  print  mail. 


.Essential 
.Nice  to  have 
_Do  not  need 


.Available 
.Not  available 
_Do  not  know  if 
available 


.Use  frequently 
.Use  occasionally 
.Do  not  use 


I  hav# -  /hav*  not _  bean  trained  on  this  capability  and 

I  would  classify  the  capability  as: 

_ Adequate  to  my  needs  _ Easy  to  use 

- Must  be  improved,  _ Difficult  to  use, 

how  so  _  how  so  _ _ 


WORD  PROCESSING  -  Provides  a  comprehensive  input  and  editing 
capability  that  allows  users  to  create,  edit  and  manipulate 
text  quickly  and  easily. 


.Essential 
.Nice  to  have 
.Do  not  need 


.Available 
.Not  available 
.Do  not  know  if 
available 


_ Dae  frequently 

_ Dae  occasionally 

_ Do  not  use 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


.Adequate  to  my  needs 
.Must  be  improved, 
how  so  _ 


.Easy  to  use 
.Difficult  to  use, 
how  so  _ 


FORM  PROCESSING  -  Allows  the  user  to  retrieve  a  standard 
form,  insert  the  information  required  and  then  store,  print 
or  forward  the  document.  The  user  may  also  create  special 
forms  for  one-time  or  repetitive  use. 


.Essential 
.Nice  to  have 
.Do  not  need 


.Available 
.Not  available 
.Do  not  know  if 
available 


.Dse  frequently 
.Dse  occasionally 
.Do  not  use 


I  have _  /have  not_  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


.Adequate  to  my  needs 
.Must  be  Improved, 
how  so  _ 


.Easy  to  use 
.Difficult  to  use, 
how  so  _ 


DOCDMENT  PROCESSING  -  Extension  of  the  basic  word  processing 
system  to  permit  the  pulling  together  of  text  from  various 
sources  to  create  a  large  document  in  a  consistent  format. 


.Essential 
.Nice  to  have 
J>o  not  need 


Available 
.Not  available 
.Do  not  know  if 
available 


_ Dse  frequently 

_ Dse  occasionally 

_ Do  not  use 


SCHEDULER  -  Determines  the  earliest  time  that  all  specified 
attendees  are  available  by  checking  their  individual 
calendar  records.  Once  all  attendees  have  confirmed  the  date 
and  time,  the  meeting  will  be  scheduled  and  an  entry  will 
appear  in  each  user's  calendar. 


.Essential 
.Nice  to  have 
Do  not  need 


.Available 
.Not  available 
.Do  not  know  if 
available 


_ Use  frequently 

_ Use  occasionally 

_ Do  not  use 


I  have _  /have  not _  been  trained  on  this  capability  and 

I  would  classify  the  capability  as: 


.Adequate  to  my  needs 
.Must  be  improved, 
how  so  _ 


.Easy  to  use 
.Difficult  to  use, 
how  so  _ 


How  do  you  rate  the  overall  responsiveness  of  the  system  ii 
operation? 


.Fast  (under  2  sec) 
.Adequate  (2-4  sec) 


.Slow  (5-10  sec) 

.Too  Slow  (10  sec  or  above) 


12.  In  what  OA  capabilities  has  response  time  been  the  slowest 
(list  slowest  first)? 


13.  On  the  average,  what  percent  of  your  day  is  spent  using  your 
OA  workstation? 


_  0-5%  (1/2  hour) 

_  6-12%  (1  hour) 

_  13-25%  (2  hours) 

_  26-37%  (3  hours) 

_  38-50%  (4  hours) 

_  Over  50% 


14.  Which  capabilities  do  you  use  the  most? 


15.  Which  capabilities  would  you  use  more  extensively  if  there 
were  more  users  on  the  system? 

— - - - -  EXAMPLES:  Electronic  Mail, 

Scheduler,  Bulletin  Board, 

2. _ _ _  etc. 


16.  What  specific  functions  or  products  do  you  use  the  system  to 
complete? 

1.  _  EXAMPLES:  Correspondence, 

Technical  Reports,  Speci- 

2.  _ _  fications.  Memoranda,  Admin¬ 

istrative  Reports  (by  name) , 

3.  _  Statement  of  Work,  Procure¬ 

ment  Requests,  etc. 


17.  In  what  way  does  the  system  assist  you  in  the  preparation  of 
these  products? 

_ Get  the  job  done  with  less  effort,  how  much  so? _ % 

_ Better  quality  of  end  product,  how  so? 

_ Fewer  typos 

_ Better  appearance 

_ More  timely 

_ Other  (Specify)__ _ 


The  following  are  general  descriptions  of  additional  OA 
functional  capabilities  that  may  be  installed  in  the  future 
on  your  OA  system.  Please  check  the  appropriate 
response/ responses. 


CORRESPONDENCE  CONTROL  -  Enables  the  tracking  of  formal 
correspondence  as  it  moves  from  one  addressee  (location)  to 
the  next.  It  requires  the  posting  of  receipt  and  forwarded 
entries  as  it  moves  from  one  point  to  another. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


PHOH.LaQQK  -  Provides  a  central  telephone  directory-like 
listing  available  to  all  users  which  includes  the  name, 
telephone  number,  organizational  code,  and  office  location  of 
all  on-base  personnel. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


PERT  -  Allows  a  user  to  construct  and  maintain  PERT  networks, 
and  do  CPM  analyses. 

_  Essential;  would  use  frequently 

_  Nice  to  have:  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


INTERACT  -  Allows  users  to  send  and  receive  messages  to  each 
other  in  a  dialogue  fashion  through  their  workstations. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tanka 


ROUTING  -  Allows  an  electronic  document  to  be  routed 
sequentially  to  other  users  according  to  an  input  specified 
or  pre-established  distribution  list. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 


If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


IMTERCQNNECT  ~  Allows  the  woricatatlon  to  be  linked  to  a 
computer  system  such  that  the  workstation  becomes  a  terminal 
on  that  system  for  programming,  data  retrieval,  etc. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


SIGMATURE  **  Allows  a  user  to  "sign”  a  document  electronically 
by  typing  in  an  alphanumeric  string  known  only  to 
himself/herself  and  the  system. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


w 
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AUDIBLE  MAIL  •  Allows  the  user  to  listen  to  a  computer 
readout  of  his  electronic  mall,  when  the  user  is  away  from 
his/her  workstation  but  has  access  to  a  touchtone  telephone. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 
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VOICE  MAIL  -  An  audio  version  of  electronic  mall.  Messages 
are  input  to  the  system  by  voice  and  recordings  are  retrieved 
by  the  addressees. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


I 

I 
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TELECONFERENCING  -  A  form  of  computer  eeseion  where  all 
attendees  are  simultaneously  connected  and  communicate  with 
each  other  In  an  Interactive  mode. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  Is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


STOPLIST  -  Points  out  all  occurrences  of  words  In  the  text 
which  have  been  prohibited. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  Is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


ARCHIVE  -  Allowa  docuoenta  to  be  filed  with  user-specified 
filing  codes,  keywords,  and  indexes,  such  as  subject, 
originator,  file  number,  serial  number  and  date.  Users  can 
retrieve  a  copy  or  restore  the  documents  to  active  (and 
modifiable)  status. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


FILE  TRANSFER  -  Addresses  and  transfers  stored  text  or  data 
files  from  one  computer  storage  system  to  another,  or  from 
one  station  (assuming  local  storage)  to  another,  etc. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


ELECTROMIC  SPREAD  SHEET  -  Allows  the  user  to  construct 
columns  and  rows  of  numbers.  Column  and  row  totals  are 
automatically  corrected  as  individual  members  in  the  matrix 
are  modified. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 


DATA  FILE  -  Allows  the  user  to  create,  edit,  maintain  and 
retrieve  data  in  a  format  (with  edit  criteria)  as  specified 
by  the  user. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


REPORT  WRITER  -  Allows  a  user  to  subject  a  data  file  to 
elementary  numeric  manipulations  and  to  retrieve  the  results 
in  a  report  format  specified  by  the  user. 

_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 

If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 

Products  Tasks 
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PS 


INTERROGATE  -  Allows  a  user  to  display  a  recordCs)  meeting 
specified  selection  criteria  from  a  data  file  specified  by 
the  user. 


_  Essential;  would  use  frequently 

_  Nice  to  have;  would  use  occasionally 

_  Do  not  need 


If  this  capability  is  made  available,  specify  which  office 
products  and  their  associated  tasks  would  be  accomplished 
more  efficiently. 


Products 


Tasks 


c 


19.  To  what  level  would  you  increase  your  usage  of  the  system  if 
it  offered  all  the  capabilities  described  above. 


0-5%  (1/2  hour) 

6-12%  (1  hour) 

13-25%  (2  hours) 
26-37%  (3  hours) 
38-50%  (4  hours) 
Over  50% 


20.  How  would  you  rate  the  reliability  of  the  hardware? 


Excellent 


Satisfactory 


21.  What  hardware  devices  have  required  the  most  maintenance? 


CRT 

Keyboard 
Printer 
Disc  Drives 

Telecom  Interface  or  Telephone  Lines 
Other  (Specify)  _ _ 


22.  If  hardware  failure  occurs,  how  would  you  rate  the  response 
time  for  maintenance? 


Adequate 


Too  Slow 


23.  Do  you  have  a  personal  contact  for  questions  and  problems 
concerning  OA  software  applications? 


In-house 

Vendor 


24.  How  often  have  you  experienced  software  problems? 


_  Rarely  (Once  every  3  months) 


_  Occasionally  (Once  a  month) 


Often  (Twice  a  month) 
With  what  capability: 


25.  How  would  you  rate  the  response  time  for  software 
maintenance? 


_  Adequate  _  Slow 


Too  Slow 


26.  How  often  does  a  system  failure  or  crash  occur? 


_  Rarely  _  Occasionally  _  Often 

(Once  every  3  months)  (Once  a  month)  (Twice  a  month) 


27.  Are  there  backup  and  recovery  procedures  in  the  event  of 
system  failures? 

No  _  Yes  _ 


28.  Do  you  have  to  repeat  (or  re-enter)  work  already  done  after  a 
system  failure  or  crash? 

No  _  Yes _ 

29.  Does  the  potential  unreliability  of  the  system  inhibit  you 
from  using  it  for  quick  response  or  fast  turn-around? 

No  _  Yes  _ 

(Please  comment) _ 


30.  Are  you  given  satisfactory  notification  of  scheduled 
downtime? 

No  _  Yes  _ 


31.  what  are  the  most  frequent  reasons  for  the  scheduled 
downtime? 

Scheduled  maintenance  _ 

Telecommunications  _ 

Software  enhancements _ 

Hardware  modifications _ 

Other  (specify) _ 


32.  Do  you  or  would  you  like  to  use  your  workstation  for  non-OA 
functions? 

No _  Yes _ 

_  Programming 

_  Data  retrieval 

_  Other  (Specify) _ 


33.  Do  you  by-pasa  the  OA  functions  and  use  your  workstation  to 
directly  Interact  with  systen  software? 


No 


Yea 


(If  yes,  why) 


'j".' 


34.  To  what  level  would  you  Increase  your  cumulative  usage  of  the 
system  If  your  workstation  accumulatively  added  these 
features: 


OSAGE 

06-12%  13-25%  26-37%  35-50%  51-62%  63-75% 
(1  he)  (2  hr)  (3  hr)  (4  hr)  (5  hr)  (6  hr) 

WORKSTATION 

WITH: 


Intelligence 
Local  Storage 
Local  Plotter 
Engln.  Graphics 
Voice 

488  Interface 


35.  What  do  you  like  the  most  about  your  OA  system? 


36.  what  do  you  like  the  least  about  your  OA  system? 


Data  Collection  Monitor; 


Received  _ 
Reviewed  . 
Tabulated. 


/S/. 
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SUMMARY  OF  SURVEY  QUESTIONS  AND  RESPONSES 


This  is  a  summary  of  the  questions  contained  in  the  NALTOACS 
sur\-ey  listed  on  pages  1  through  20.  This  sunmary  examines  each  survey 
question  to  determine  whether  the  responses  in  each  question  lists  a 
response  judged  positive  (by  the  author)  for  OA  in  general  or 
negative/neutral  for  OA  in  general. 


'Question 


1  -  9: 


10: 


11 

12  -  16 

17 

18 


19 

20 


21 


22 

23 

24  -  26 


27 


28  -  29 


negative/neutra 1 

Series  of  13  OA  systems  functions  the  respondent 
is  asked  to  judge.  Each  of  the  functions  has  six 
response  areas;  five  of  these  areas  list  positive 
responses  for  OA  first.  Therefore,  there  are  65 
positive  and  13  negative/neutral 

positive 

negative/neutra 1 

positive 

Series  of  17  additional  OA  functional 
capabilities  that  are  available.  Each  functicn 
has  one  positive  response.  Therefore,  there  are 
17  positive  responses. 

negative/ neutra 1 

positive 

negative/neutra 1 

positive 

negative/neutra 1 

positive 

negative/neutra 1 

positive 


30  -  34:  negative/neutral 

35:  positive 

36:  negative/neutral 

The  NALTO/^S  questionnaire  provides  the  opporfanity  for  124 
responses.  Of  that  124,  92  of  the  responses  are  judged  by  the  author 
to  be  positive  (in  favor  of  OA  in  general)  or  the  first  response 
cption  listed  is  positive.  The  remaining  32  are  considered  negative  or 
neutral  of  OA  in  general  or  the  first  response  option  listed  is 
negative/neutral.  Dividing  the  number  of  positive  responses  (92)  by 
the  total  number  of  responses  (124)  yields  a  figure  of  74%. 


APPENDIX  B 


PROJECTIONS  OF  POTENTIAL  SAVINGS  THROUGH  THE 
INCORPORATION  OF  OFFICE  AUTOMATION  EQUIPMENT 
AT  NAVAL  WEAPONS  CENTER  CHINA  lAKE,  CALIFORNIA 

Table  1,  Table  2,  and  Table  3  are  graphic  illustrations  of 

estimates  made  by  the  Maddox  Group  to  illustrate  the  opportunities  to 

automate  various  office  functions  at  IvWC  China  Lake  (Maddox,  February 

1981) . 

Pages  73  through  77  of  this  Appendix  are  taken  from  the  ATI PS  Concept 
Development  Paper  (NWO,  Nferch  1982). 
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(The  figure  in  the  "after"  column  refjresents  tlie  project nl  [ercentag 
of  ctiange  in  the  hours  sjient  on  each  activity  after  autcjiiat  ion.  ) 


Calls 


Personnel  Costa.  Direct  costs  are  the  salaries  of  personnel 
required  to  operate  the  ATIPS.  The  salary  schedule  for  a  technician  is  averaging 
$25,951  annually.  For  the  purposes  of  this  economic  analysis,  the  figure  of  $25,951 
was  inflated  by  9  percent  to  account  for  cost  of  living  increases  between  FY81  and 
FY82,  adjusted  by  10  percent  to  account  for  cost  of  living  increases  between  FY82 
and  FY83,  then  adjusted  by  31  percent  to  conform  to  the  guidelines  set  forth  by  the 
A-76  Cost  Comparison  standards. 

Average  salary  for  FY81  “  $25.9  K 

Average  salary  for  FY82  »  $28.3  K 

Average  salary  for  FY83  =  $31.1  K 

Adjustment  prescribed  by  A-76  31  percent 

Total  adjusted  average 

technician  salary  =  $40.7  K 

The  adjusted  salary  of  $40.7  K  is  used  as  the  salary  of  the  ten 
personnel  who  are  required  for  the  operation  of  ATIPS.  This  staff  will  be  used  to 
operate  and  maintain  the  system,  perform  restart  and  recovery,  and  if  time 
permits,  training  and  helping  users  become  familiar  with  ATIPS. 

2.  Maintenance  Costs.  Maintenance  costs  will  not  be  incurred  in 
FY83,  since  it  is  assumed  that  the  first  increment  of  the  ATIPS  will  not  be 
functional  until  FY84.  Monies  will  be  committed  during  the  fiscal  year  requested, 
but  delivery  of  hardware  will  be  staged  over  time  to  facilitate  installation  and 
training.  Annual  maintenance  costs  are  predicated  on  estimated  maintenance 
contract  costs  of  $250  per  terminal,  $5,000  per  processing  unit,  and  $10,000  per 
shared  resource.  The  first  increment  of  ATIPS  (FY84)  will  require  60  terminals, 
three  processors,  and  one  OCR  for  a  maintenance  cost  of  $40,000.  Costs  in  FYOS 
reflect  the  use  of  an  additional  480  terminals,  10  processors,  one  OCR,  and  one 
printer  for  an  annual  cost  of  $230,000.  Costs  for  PY86  include  the  third  increment 
of  540  terminals,  11  processors,  one  OCR,  and  one  printer  at  an  annual 
maintenance  cost  of  $440,000.  Costs  for  FY87  include  the  fourth  and  final 
increment  of  540  terminals,  11  processors,  one  OCR,  one  quality  printer,  and  one 
video  disc  at  an  annual  maintenance  cost  of  $660,000. 

3.  Other  Costa.  Other  costs  include  the  annual  costs  for  supplies 
(paper,  ribbons,  printwheels,  etc.),  equipment  and  building  rental,  if  applicable,  and 
for  utilities  (electric,  heat,  etc). 

4. 2. 5.2  ATIPS  Benefits.  Benefits  are  both  quantitative  in  dollar  savings  and 
qualitative  in  terms  of  a  more  timely  higher  quality  product. 

a.  Quantitative  Benefits.  Quantitive  benefits  are  derived  from 
calculating  dollar  savings  attributed  to  the  more  efficient  operation  provided  by 
ATIPS,  resultant  cost  avoidance  savings  and  reduced  expenditures  of  the  current 
system. 

1.  The  results  of  a  requirements  analysis  performed  by  The  MATTOX 
Group  in  December  1980  through  February  1981  at  NWC,  shows  that  ATIPS  will 
result  in  a  20  percent  savings  in  time.  This  savings  will  result  in  additional  tine 
being  made  available  to  a  staff  member  to  work  on  projects  and  is  converted  to 
cost  savings  by  multiplying  the  20  percent  factor  against  the  persons  ■salary 


(including  all  benefits).  This  value  agrees  with  information  from  other  similar 
office  automation  projects  from  both  Government  and  industry  and  is  felt  to  be 
conservative.  Increases  in  efficiencies  far  above  this  20  percent  value  have  been 
reported  in  well  organized  and  scheduled  office  automation  activities.  Industry 
publications  indicate  typical  productivity  increases  of  from  10  percent  to  30 
percent  using  av:;ilable  systems.  Amoco  Production  Research  Co.  claims  a  60 
percent  increase  in  the  productivity  of  its  personneL  Booz,  Allen,  and  Hamilton 
estimates  a  20  percent  increase  in  productive  time  with  the  adoption  of  new  office 
automation  technology  with  the  greatest  payoff  coming  to  senior  managers  in  the 
form  of  enhanced  decisionmaking.  For  the  purpose  of  calculating  cost  avoidance 
based  on  the  20  percent  savings,  an  average  weighted  salary  (average  of  all  salary 
classes  including  benefits  in  FY83  dollars)  of  $43.8  K  is  used. 

2.  This  value  is  obtained  by  proportionally  averaging  the  number  of 
members  in  each  Job  category  times  their  salaries.  These  salaries  for  each  Job 
category  are:  Scientist  and  Engineer  ($35,241),  Technicians  ($25,950),  Clerical 
($14,077),  and  Administrative  ($26,634).  The  resultant  average  of  $27,876  is  then 
inflated  by  9  percent  to  account  for  cost  of  living  increases  between  FY81  and 
FY82,  and  by  10  percent  for  FY82-FY83  increases,  and  adjusted  by  31  percent  to 
conform  to  the  guidelines  set  forth  by  the  A-76  Cost  Comparison  Standards  for 
benefits. 

Average  salary  for  FY81  *  $27.9  K 
Average  salary  for  FY82  *  $30.4  K 
Average  salary  for  FY83  *  $33.4  K 
Adjustment  prescribed  by  A-76  31  percent 


Total  adjusted  average 
salary  * 


$43.8  K 


3.  In  FY85,  it  is  cjtimated  that  65u  employees  will  potentially  be 
affected  by  ATIPS  from  bringing  on  line  540  terminals  and  will  realize  an  increase 
in  efficiency  of  20  percent  at  a  cost  avoidance  savings  of  $5.7  M.  This  estimate  of 
the  relationship  of  employees  to  terminals  is  based  on  the  known  synergistic 
effects  that  come  about  through  social  interaction  between  employees  (e.g., 
roommates  sharing  a  terminal  exchange  information  to  the  gain  of  each).  In  FY88 
an  additional  950  personnel  will  potentially  be  affected  by  .\TIPS  from  adding  an 
additional  540  terminals.  This  results  in  an  additional  savings  of  $8.3  or  a 
cumulative  annual  cost  avoidance  savings  of  $14  M.  In  FY87  an  additional  540 
terminals  will  be  added  providing  use  to  an  additional  900  personnel.  This  results  in 
an  additional  savings  of  $7.9  M  or  a  cumulative  annual  cost  avoidance  savings  of 
$21.9  M.  This  annual  savings  will  be  the  same  for  the  remaining  5  years  since  no 
further  terminals  are  anticipated  after  FY87  as  a  part  of  this  purchase.  Labor 
savings  due  to  cost  avoidance  over  the  first  8  years  of  ATIPS  in  constant  FY83 
dollars  is  $151  M.  These  savings  are  broken  out  for  each  year  over  the  system's  life 
cycle  and  are  shown  in  Figure  4-7. 
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TOTAL  DISCOUNTED  307  5,5^0  12,119  17,186  15,628  U,190  12,896  11,601  10,667  100,13<i 
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4.  Other  dollar  savings  can  be  achieved  through  a  reduction  in  the 
travel  budget  made  possible  by  the  enhanced  communications  of  ATIPS,  reduction 
in  the  current  maintenance  costs  of  the  status  quo  system,  and  reductions  in  costs 
for  the  Center  distribution  system  (vehicle  maintenance  and  operations  costs).  A 
breakout  of  these  potential  cost  savings  in  FY83  dollars  shown  below  are  as 
follows: 

COST  SAVINGS  ELEMENT  LIFE-CYCLE  COST  SAVINGS 


I 


.■S 

i 
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1.  Off  Center  Travel  $8,820  K 

2.  Maintenance  Costs  of 

Office  Equipment  $4,200  K 

3.  Off  Center  Travel  Time  $1,470  K 


5.  The  above  savings  in  travel  are  attributed  to  an  estimated  IS 
percent  reduction  in  traveling  requirements  made  possible  by  the  enhanced 
communications  of  ATIPS,  allowing  S&E's  to  communicate  with  off-Center 
personnel,  data  bases,  and  libraries,  etc.  NWC  averages  7,000  trips  per  year  in  off- 
Center  travel  at  an  average  cost  per  trip  of  $1,200.  The  study  estimates  that 
approximately  IS  percent  of  this  travel  can  be  saved  each  year  through  the  use  of 
ATIPS.  This  amounts  to  $420  K  the  first  year  ATIPS  is  fully  operational.  Using 
FY83  dollars,  the  life-cycle  (8  years)  cost  savings  realized  amount  to  $8,820  K. 

6.  It  is  expected  that  savings  will  result  from  a  reduction  of  time 
spent  in  a  travel  status.  Assuming  one  day  of  lost  time  for  each  of  the  trips 
avoided  above  the  savings  the  first  year  would  be  $70,000  and  $1,470,000  for  the  8 
years. 

7.  Savings  of  the  current  maintenance  results  from  decreased  wear 
through  less  usage  on  such  items  as  copiers,  typewriters,  and  other  office 
equipment.  During  FY  80,  for  example,  NWC  spent  $260  K  to  maintain  office 
equipment,  $20  K  per  month  on  copy  machine  paper  and  $600  K  on  copy  machine 
rentals,  in  the  same  period,  $110  K  worth  of  new  word  processing  equipment  was 
purchai,ed  and  $139  K  was  spent  to  lease  v/ord  processing  equipment.  These  figures 
were  converted  to  FY83  dollars  and  it  is  assumed  that  SO  percent  of  the  related 
cost  can  be  avoided. 

8.  Additionally,  it  is  expected  that  there  will  be  a  cost  avoidance  of 
$322  K  (FY83  dollars)  in  FY84  in  that  the  Center  will  defer  anticipated  purchases 
of  office  supplies  and  equipment  to  wait  for  ATIPS  to  supply  the  necessary 
capability.  The  savings  due  to  cost  avoidance  over  the  expected  8-year  life  cycle 
of  the  ATIPS  project  total  is  $166  M,  and  is  plotted  by  installation  phase  in 
Figure  4-8.  This  graph  shows  the  acquisition  costs  and  the  total  cost  savings  as  the 
equipment  is  installed  and  becomes  operational.  A  very  early  break  even  point  (3 
months)  is  shown.  Hov/ever,  a  payback  period  of  1.25  years  (the  inflection  point  in 
the  cost  savings  line)  is  claimed  based  on  the  sensitivity  of  the  estimated  20 
percent  productivity  index  rate. 
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